Hacker News Reader

Comment by user5994461 | Hacker News Reader

Good read, except the part where the author says there are no existing solutions for processing logs. There are quite a few robust scalable ones.

syslog-ng, logstash or fluentd on the host to collect and aggregate logs. (logstash/fluentd can parse text messages with regex and handle a hundred different things like s3/kafka/http but they are much more resource intensive).

kibana or graylog to centralize logs and search, the storage is elasticsearch.

A simple syslog-ng on the devices could probably do the job. Little known fact about syslog, it can reliably forward messages over TCP, logs are numbered, have retries and syslog-ng can do DNS load balancing.

View on HN

Yeah, fluentd... I had to look into its in_stdin plugin, and I can tell you one thing: it simply doesn't handle regexes properly. In fact, none of its input plugins can correctly handle the situation where a multiline regex doesn't match the whole line, unless they do what in_tail did (split the lines yourself and disallow the multiline regexes).

Edit: Sorry, I meant "Fluent bit". No idea how fluentd handle this scenario, but I was told it was too slow (being written in Ruby) so that's why the switch to fluentbit was made.

Last time I tried, fluent-bit did not support IPv6. Maybe it does now? (Maybe it does not: https://github.com/fluent/fluent-bit/issues/2056 has been opened recently). Anyway, compared to the solution in the article, it seemed way too fanciful.

Fluentd would not just be slow, but also run out of memory. I am no ruby-head, but a then-colleague of mine helped configure it. It still ran out of memory. I had no patience for a log system that did not work out of the box on one computer basically just logging failed ssh login attempts.

I've done multiline processing with logstash. It works but it's not necessarily efficient or simple. Depends a lot on the pattern and whether the multiline stack traces you want to extract turns out to be 700 lines long sometimes (yes, speaking from experience).

I'd personally recommend to not bother with processing multiline output into a single message. Lots of trouble for no benefits. It's just a stream of lines at the end of the day, it will look the same in tail and kibana.

Checkout Vector. vector.dev

GitHub project here https://github.com/timberio/vector/

The flexibility of Fluentd and the community around it are great, but ultimately the resource intensity proved to be too high and too unpredictable for our use case.

We ended up building our own general purpose log parser in Go to support our hosted log monitoring platform. Though it's still somewhat young, performance is great (beats Fluent Bit in most of our benchmarks), and it's nearly as flexible as Fluentd in terms of configuration. If you want to check it out, we recently open-sourced it and are always looking for more feedback: https://github.com/observIQ/stanza

There's also good ol' dirty syslog.

elasticsearch would not cope with that volume of data on the hardware described.

Well there is clickhouse. And projects like this https://github.com/flant/loghouse

Well, the article doesn't mention the hardware. Only one machine for 5 TB a day which doesn't add up as far as I am concerned.

If they were storing everything in S3, it's possible to do something similar with ElasticSearch for the same order of costs (maybe three times?), the money going to EBS storage instead. ElasticSearch allows to query and visualize logs, which plain S3 storage doesn't, it's worth a bit more IMO.

I think the author's math tracks. Our pipeline looks something like

hosts -> rsyslog collector -> kafka -> custom crunching -> elastic

which is a bit inefficient and we try to go hosts -> kafka when we can but a lot of stuff only supports rsyslog so it's there and simple enough.

The rsyslog collector and our crunchers are teeeny tiny compared to the rest of the pipeline and can chew through up to a week of backlog in a few hours. The bottleneck is the network for us and if we upped the pipe to 10G we could probably get away with a single host.

I missed where a volume of data was described that was outside elastic search's capacity. Seems unlikely that it "would not cope" regardless of configuration.

Reply on news.ycombinator.com