Show HN: Dendrite, a golang log parser
github.com
github.com
"Some might argue that this test is unfair to the backtracking implementations, since it focuses on an uncommon corner case. This argument misses the point: given a choice between an implementation with a predictable, consistent, fast running time on all inputs or one that usually runs quickly but can take years of CPU time (or more) on some inputs, the decision should be easy. Also, while examples as dramatic as this one rarely occur in practice, less dramatic ones do occur. Examples include using (.) (.) (.) (.) (.*) to split five space-separated fields, or using alternations where the common cases are not listed first. As a result, programmers often learn which constructs are expensive and avoid them, or they turn to so-called optimizers. Using Thompson's NFA simulation does not require such adaptation: there are no expensive regular expressions."
Since the Go Regexp implementation trades off speed for predictability in performance, I'm wondering if in the real world the performance hit is noticeable.
Dendrite is actually a port of an unreleased library I wrote in C with pcre. For us, the slowdown (which at our log velocity of hundreds of lines per second per box, still looks like 0% cpu in top), was well worth the productivity we got from moving to Go.
Why did you use yaml for configs? Yaml is/can be kind of fragile (whitespace, etc). Do you consider something more practical - nginx config formats?
How to set-up log collector? For example I need to collect logs from several instances. Could not find in cookbook.
How do you separate logs for different hosts/applications?
How do you ship logs? UDP? TCP? TLS?
How is this compared vs Logstash?
Thank you!
Yaml is the most common reasonably suitable format for things that look like config files. It's familiar to more developers than nginx config, therefore we hope more people can contribute.
> How to set-up log collector? For example I need to collect logs from several instances. Could not find in cookbook.
Did you look at the tutorial? https://github.com/onemorecloud/dendrite/blob/master/tutoria...
> How do you separate logs for different hosts/applications?
Right now, the output map has a couple automatically inserted keys for hostname, shipping timestamp, and the name of the application. This isn't terribly well-documented.
> How do you ship logs? UDP? TCP? TLS?
Yes ;)
> How is this compared vs Logstash?
Dendrite isn't trying to be Logstash. I'd like dendrite to be the agent you use with logstash, greylog, papertrail, or whatever. I think the agnostic nature of dendrite will be a big win, as dev/ops people can persist their data to multiple stores, or swap out stores more easily. The application we wrote Dendrite for currently persists logs to three different stores.
> Yes ;)
ok, missed it, checked again. UDP, TCP - yes. TLS - not yet?
I checked tutorial, nothing there. Since most [semi]serious projects have a least some servers, I suggest to add a separate example, smth like:
1. We have 5 servers. And we collect logs to server 6. 2. These are settings you need to use for 5 servers. 3. And these are settings you need to use for the collector [IP:Port:TCPwithTLS].
> Dendrite scrapes your existing logs
Does Dendrite tail? Can Dendrite consume/scrape some existing/old log files?
BTW, what was your actual max throughput? Did you use some internal queue in the design, like rsyslog is doing?
> Dendrite isn't trying to be Logstash. I'd like dendrite to be the agent you use with logstash, greylog, papertrail, or whatever.
I am confused here a bit. So Dendrite is like StatsD on steroids?
tcp + tls is a one-liner I haven't added
> Does Dendrite tail? Can Dendrite consume/scrape some existing/old log files?
Yes, dendrite tails/follows, with optional backfilling.
> Did you use some internal queue in the design, like rsyslog is doing?
Since you're using existing logs, all we do is maintain a pointer into the logs on disk. When we start consuming more transient data (e.g. perhaps we poll jmx for you), we'll have to add the queue.
> I am confused here a bit. So Dendrite is like StatsD on steroids?
Dendrite is "tail -f *.log | convert_to_json.pl | nc" on steroids.
Most of the other log tailer/shippers are in java, so untenable for RAM-constrained environments. Dendrite also has an emphasis on structuring and re-emitting logs in e.g. json, rather than say, matching specific lines and sending email when you see them, or just forwarding all logs in their original format.