LogZoom: A fast, lightweight substitute for Logstash/Fluentd in Go

To respond to your point about absence of Grok like facility, avoiding the need to unmarshal and remarshal the data while passing through LogZoom was an explicit design requirement. The blogpost refers to our pain with Logstash/Fluentd etc. We were in a situation where our production code was fighting for resources against a log collecting facility.

In general, it's best to process the data (to the extent possible) closest to it's point of origin. It's orders of magnitude cheaper to create a well structured log line straight from your production code (where it's just some in-memory manipulation of freshly created strings) rather than in a post-processing step inside a separate process (or machine).

I've spent years dealing with performance problems in global scale production stacks and a surprisingly high number of resource bottlenecks (memory/CPU/Disk IO) etc. are caused by ignoring this simple principle.

I've lost count of the cases where a simple restructuring of the architecture to avoid a marshal/unmarshal step drastically cuts down resource requirement and operational headaches. Unfortunately a whole lot of industry "best practices" (exemplified by the Grok step in Logstash) encourage the opposite behavior.

Is this unfair? Perhaps I misunderstood the docs.

If you don't need to curl|sh from a .io (or .sh) domain to install it, it's not worth using apparently.

I wouldn't build a monitoring or inventory system on rsyslog, but I don't hesitate to use Fluentd. rsyslog was intended for logs only, and using it in any other way seems an abuse, even if smart and somewhat fitting.

I haven't used logstash, but I bet it operates in a similar way on its data sink border.

It can also queue up messages in memory and/or to disk if your remote data sink is having a hiccup.

And for those wondering how to send multiline data, well you don't. If you need to write out a big blurb, you write it out on a single line from the application. If using structured data, you can output lines as separate array items. The current built-in limit for a syslog line is 8096 bytes, but that's tunable. Just make sure the thing that writes to syslog doesn't have a low hardcoded limit like older versions of logger from util-linux (1024 bytes)

edit: the version of rsyslog shipped with the distros might be a bit dated. They're providing packages for their latest stable version, we're using that and it works pretty well.

We have it set up to write logs locally (with a limited rotation) as well as forward them via TLS to a central Rsyslog server that collects the log in a single tree with a much longer retention time. (We don't use any of the non-file outputs, but we do sync to S3 for archival.)

It has major issues. For one, its spooling implementation is flaky. /dev/log is a limited, synchronous-blocking FIFO buffer, which means that everything that logs (including OpenSSH!) will choke if the buffer is full. For some reason, just a tiny bit of packet loss will throw Rsyslog.

It also frequently is unable to recover from a network blip, and a restart is the only solution. But its spool file is badly implemented, so on restart it will typically ignore the old spool files and start anew — meaning you lose data. Someone wrote a Perl script to fix a broken spool directory, but I never got it to work.

Ironically, Rsyslog is also terrible at logging what it's unhappy about at any given time, so whenever something bad happens, you probably won't get anything in the system log.

Rsyslog's configuration is a curious beast, and by curious I mean infuriating. Rsyslog originally had an antiquated, ad-hoc and messy line-oriented configuration file format (with directives like "$RepeatedMsgReduction off"), and author decided to transition to a more modern, block/brace-based syntax. Unfortunately, he decided to do this gradually, and both syntaxes can co-exist in the same file. For a while, many of the options were only available in the old syntax, so you had to mix the two.

Which leads me to the next problem: The documentation is absolutely atrocious. The Rsyslog site is a fragmented mess of mostly outdated information. It's gotten better with v8, but it's still the worst OSS project documentation I've encountered. There's no reference section that lists the possible config options. Frequently there is no documentation for a particular setting. Rsyslog is quite finicky about some combinations of options (like TLS driver configs) and you have to proceed by trial and error. Frustratingly, it will silently ignore some config errors (such as trying to set up multiple TCP listeners, which is still not supported).

The new config format is better, but it still has the feel of something that has been implemented before it was fully designed.

Again, we don't use any of the fancy output modules. Maybe they are solid, but based on my experiences with the simple file-based stuff, I wouldn't bet on it.

As an aside, it's worth pointing out that Rsyslog is still using the Syslog protocol, which has all sorts of issues (not consistently implemented by clients or servers; does not support multi-line messages). Rsyslog has another protocol, RELP, that I believe you can use for forwarding, but I don't think it's been implemented outside of Rsyslog.

As far as I can tell, there aren't any good alternatives. syslog-ng's forwarding support is commercial and quite expensive. Logstash might work, but I don't want to run a memory-hungry Java app on each box.

Right, we considered using multiple Logstash processes, but we really didn't want to run three instances of Logstash requiring three relatively heavyweight Java VMs. The total memory consumption of a single VM running Logstash is higher than running three different instances of LogZoom.

We looked at the Filebeat Redis output as well. First, it didn't seem to support encryption or client authentication out of the box. But what we really wanted was a way to make Logstash duplicate the data into two independent queues so that Elasticsearch and S3 outputs could work independently.