Logstash joins Elasticsearch
elasticsearch.com
elasticsearch.com
There's a neat demo of Kibana here: http://demo.kibana.org/#/dashboard/elasticsearch/Logstash%20...
The only thing that isn't fully baked in with this stack is alerts (e.g. sending an email if a certain error log message comes in), but you can do that using Logstash filters and outputs, although there's no pretty UI.
There are some excellent Chef cookbooks for setting up Logstash and friends too:
- Logstash: https://github.com/lusis/chef-logstash
- Elasticsearch: https://github.com/elasticsearch/cookbook-elasticsearch
- Kibana: https://github.com/lusis/chef-kibana
https://github.com/bpaquet/node-logstash
It is "logstash compatible" (at ElasticSearch, so it works with Kibana) and in my experience very easy to work with, and probably a lot lighter weight than the JRuby version.
What's most interesting is that Elasticsearch seems like a completely open source (and widely used) offering of a product that Splunk charges close to oracle pricing for.
Shameless plug: If you're looking for an opportunity at a well-funded true real-time analytics company in silicon valley...feel free to ping me. There's lots of exciting and fun work to do in this area.
It turns out that lucene based systems are pretty good at information retrieval and aren't shackled with all the OLTP requirements most databases have.
I needed 12 ES boxes for every one Splunk box to handle the 100MB/day log load of my system, and even then they ran at a high load and searches often failed, and in some cases it took hours for the indexer to catch up.
As mentioned in another comment in this post, I was doing 300gigs of data per day with an elasticsearch cluster size of 7 elasticsearch nodes (16 cores & 16gb ram per node) and load was around 5-10% cpu utilization.
100MB/day is pretty small in terms of log data, I think. If you attempt this again, please invoke the community (elasticsearch's is great!) and see if we can assist you in figuring out what's busted.
hekad is written in go which compiles down to native code while logstash is written in jruby which is not the most performant runtime.
On my servers I use the open source version of nxlog to collect various logs and forward them to a central nxlog server, which in turn feeds logstash. Behind logstash I have configured elasticsearch as storage and I use kibana as a GUI to search and browse.
Did you find a usable GUI or web interface to view the logs?
we found that when using logstash even just for pushing stats to statsd it was not performing well enough. i've experimented with hekad pushing to elasticsearch on our staging cluster and performed well enough but we had weird problems showing up in nagios when we were using logstash+elasticsearch in production (checks were timing out even though we were seeing no degradation of performance on the servers). because of this it is quite difficult to get any kind of central log pushing into production. :(
The next release of logstash (1.2.0 is in beta) has a 3.5x improvement in event throughput. For numbers: on my workstation at home (6 vcpu on virtualbox, host OS windows, 8gb ram, host cpu is FX-8150) - with logstash 1.1.13, I can process roughly 31,000 events/sec parsing apache logs. With logstash 1.2.0.beta1, I can process 102,000 events/sec.
Processing speed will vary greatly by what you are doing with your events and it doesn't make sense to generalize performance characteristics globally, especially with a metric that, alone, doesn't really tell me much (cpu utilization).
If it's slow, it's a bug. We can fix it. :)
Further, you can use hekad with logstash and with elasticsearch (one or both together, it doesn't matter).
In terms of problems solved, logstash helps solve transport and real-time processing problems. In cases where the logstash agent is too resource intensive, the logstash community offers many alternatives on this site: http://cookbook.logstash.net/recipes/log-shippers/
The community (myself included) is very interested in helping logstash be a success for its users, so if you do see performance problems, things that behave weirdly, or anything strange, it's probably a bug, and we can fix it.
The short version of all of this is captured by the project principles listed in the logstash readme: https://github.com/logstash/logstash/#project-principles
<3
Honestly, I've never heard of either company, although I obviously wish them the best of luck. Am I just out of touch?
Except for grep of course.
Of the three, you should at least could have heard of ElasticSearch: it is a general purpose search server based on Apache Lucene like its cousin Apache SOLR (maybe you know it), but with REST API, JSON support etc..
You can use it for searching stuff and one (but not the only one) scenario is logging. In this case you use some other software (like Logstash but I think there are others too) to collect logs from different sources (i.e. syslog for operating systems, ruby apps via gems, raw tcp etc..) and according to a friend of mine that uses it for this purpose it's very good for its speed, easy to use, to scale etc..
N.B. I invite anyone more expert to elaborate/correct what I've said, I've just used Apache SOLR in the past and never tried ES
Logstash is a sort of pipeline for data you want to log: you can define multiple inputs, transform/filter the data, and then define multiple outputs.
Example 1: read in your apache logs (input), attach geoip data (transform), and then push the resulting data to elasticsearch (output).
Example 2: read from syslog (input), grep on the input to ignore certain files (filter), then push to graphite for graphing (output).
you can have multiple inputs, multiple transforms/filters, and multiple outputs. You can also chain logstash instances together, so you can have "roll up" logs. Logstash itself is a bit heavy in terms of CPU/RAM (it is written in Java), so there are a few, lighter weight "shippers", and you can ship into a Redis instance to proxy events.
Elasticsearch is a java-based search engine with a great REST API and a _lot_ of features. It is built on top of Lucene. It doesn't have a built in GUI. It also scales out super easily.
Kibana is front-end to Elasticsearch, which lets search/visualize your log events.
Ok, those are the parts, why this is interesting: As other commenters have pointed out, this is a powerful combination for understanding your log data. You can ship logs from apps, services and hosts, visualize what's going on, search, correlate, etc.
http://www.google.com/finance?q=splunk&ei=VwsdUqi2L5qglwON4g...
They are important, hard to do well and have a bad habit of only causing issues which swallow engineering time when you are firefighting furiously trying to scale core services.
That's why as someone pointed out separately Spunk is a $5Bn company and people who have had these problems previously are very excited by this news.
(It's also why StatsD&Graphite/OpenTSDB, Riemann/Sensu etc etc are all super interesting)
Looking forward to some really tight integration between the Logstash, ES and Kibana.
Since you can access the logs via the Elasticsearch API, we made users' recent logs available to them in our dashboard: https://swiftype.com/blog/api-logs.html
Those guys are meant to be releasing a new re-vamped version at the end of October, from the screenshots and videocasts, looks pretty good:
The event rate going into this cluster was about 5000 events/sec on average (burst up to 10,000 events/sec sometimes).
During a maintenance (two machines going offline for disk repairs), I benchmarked the surviving 5-node cluster at 88,000 events/sec peak performance.
In terms of capacity planning, this means that we could have a 9x increase in normal event load and still not need to grow the cluster's processing capacity.
Persistent storage is another story. We stored about 300GB/day of events, getting us roughly 45 days of data retention before we would run out of space (2TB * 7 nodes / 300gb/day; roughly 45 days). I'm working on improving storage efficiency of logstash and elasticsearch, too, so retention should improve greatly in the long term.
For other experiences, it's useful to invoke the community and ask what others are done - the #logstash irc channel on freenode is very active as is the logstash-users@googlegroups.com mailling list.
Hope this helps!
In practical terms, and by way of example, a plain text apache access log, fully parsed by logstash (breaking out fields, etc), has historically bloated by quite a bit (6.2x I have measured). Lately, however, with improvements to logstash, better default settings, and elasticsearch being awesome, the 'inflation' number gets down to something more like 1.5x - which isn't bad considering all the awesome you get with it.
Long term, I am working towards making the 'raw data to stored data' ratio something less than 1x.
You can see some experiments I did a year ago on this: https://github.com/jordansissel/experiments/blob/master/elas...
I will repeat these experiments after the next release of logstash, and I expect storage ratios to improve significantly.
The initial event and any subsequent ones tend to arrive close together in time, so the challenge is to find something that can handle a high insertion rate, a relatively low update rate, while providing fast aggregations suitable for charting in a web-frontend. In Riak, Couchbase or HyperDex we'd use a secondary index and do our own math, but Elasticsearch is attractive because it appears to support the kind of queries we're interested in out of the box, in addition to having a good reported write-rate.
Persistence is less of an issue, because after a short period of time (a couple of hours) we would summarise the events into our analytics DB (Infobright) and so we could set a TTL on the data stored in Elasticsearch.
Again, thanks for the response and I'll check out the mailing-list and IRC channel.
Edit: Grammar
Congrats :)
I evaluated elasticsearch for RT analytics. It works wonders for point queries, where your result set is going to be small. Didn't work well for aggregate queries which need to scan lot of data. The biggest problem was field cache in Lucene. Almost all our queries needed to fo faceting which had a big impact on field cache.
Also, I don't know about Riak, but in ES the joins you can do are very limited.
If your data is read-only then Cloudera Impala is worth a try. It's really fast.
Also we've suffered data loss on several occasions with elasticsearch. This has been getting better but is still a concern. Having the external long term datastore and a handy import method made these small hiccups.
Lastly IMO elasticsearch works best, particularly when working with log like data, as a rolling window view into the data. This keeps your elasticsearch cluster to a reasonable size (saving $$) while keeping the ability to re-load old data for exploration when you need it.
[1] http://ruilopes.com/elasticsearch-setup/
[2] http://www.elasticsearch.org/guide/reference/modules/discove...
Like I always say, if it's hard to use or appears to have major flaws or pains, it's a bug, and we can fix it. Let us know! :)