Why did you use yaml for configs? Yaml is/can be kind of fragile (whitespace, etc). Do you consider something more practical - nginx config formats?
How to set-up log collector? For example I need to collect logs from several instances. Could not find in cookbook.
How do you separate logs for different hosts/applications?
How do you ship logs? UDP? TCP? TLS?
How is this compared vs Logstash?
Thank you!
Yaml is the most common reasonably suitable format for things that look like config files. It's familiar to more developers than nginx config, therefore we hope more people can contribute.
> How to set-up log collector? For example I need to collect logs from several instances. Could not find in cookbook.
Did you look at the tutorial? https://github.com/onemorecloud/dendrite/blob/master/tutoria...
> How do you separate logs for different hosts/applications?
Right now, the output map has a couple automatically inserted keys for hostname, shipping timestamp, and the name of the application. This isn't terribly well-documented.
> How do you ship logs? UDP? TCP? TLS?
Yes ;)
> How is this compared vs Logstash?
Dendrite isn't trying to be Logstash. I'd like dendrite to be the agent you use with logstash, greylog, papertrail, or whatever. I think the agnostic nature of dendrite will be a big win, as dev/ops people can persist their data to multiple stores, or swap out stores more easily. The application we wrote Dendrite for currently persists logs to three different stores.
> Yes ;)
ok, missed it, checked again. UDP, TCP - yes. TLS - not yet?
I checked tutorial, nothing there. Since most [semi]serious projects have a least some servers, I suggest to add a separate example, smth like:
1. We have 5 servers. And we collect logs to server 6. 2. These are settings you need to use for 5 servers. 3. And these are settings you need to use for the collector [IP:Port:TCPwithTLS].
> Dendrite scrapes your existing logs
Does Dendrite tail? Can Dendrite consume/scrape some existing/old log files?
BTW, what was your actual max throughput? Did you use some internal queue in the design, like rsyslog is doing?
> Dendrite isn't trying to be Logstash. I'd like dendrite to be the agent you use with logstash, greylog, papertrail, or whatever.
I am confused here a bit. So Dendrite is like StatsD on steroids?
tcp + tls is a one-liner I haven't added
> Does Dendrite tail? Can Dendrite consume/scrape some existing/old log files?
Yes, dendrite tails/follows, with optional backfilling.
> Did you use some internal queue in the design, like rsyslog is doing?
Since you're using existing logs, all we do is maintain a pointer into the logs on disk. When we start consuming more transient data (e.g. perhaps we poll jmx for you), we'll have to add the queue.
> I am confused here a bit. So Dendrite is like StatsD on steroids?
Dendrite is "tail -f *.log | convert_to_json.pl | nc" on steroids.