What does a modern metrics stack look like? There are so many words... Graphfana, Kibana, StatsD, Graphite, etc. Which bits should I choose and attach together?
Not so much for the design (oh look it's white instead of black, who cares), but the way they've handled it subsequently.
The 3.x -> 4.x transition for Kibana left a product that was missing really basic features (like, the ability to set graph colors for one - a bug/feature request that's been open since last january).
As it stands, upgrading from Kibana 3 to Kibana 4 is a step backwards. You lose functionality rather than gaining it.
They also decided to require a major version bump to Elasticsearch (to 2.x) with a point release of Kibana.
I used to be super optimistic about the Elastic guys, but some of these decisions are just head-scratchingly awful.
This might be derailing the thread a bit, but is there any log management platform like ELK or Splunk that has an expressive and versatile query language like Splunk's? My biggest issue with ELK is that analytics is mostly expected to be done through Kibana's GUI, while with Splunk you can craft terse queries to do almost any sort of transformation and visualization imaginable. I don't like how ELK is so GUI-oriented.
https://www.elastic.co/guide/en/elasticsearch/reference/curr...
There is a Python implementation that makes creating complex queries pretty easy:
https://github.com/elastic/elasticsearch-dsl-py
I agree with the criticisms of Kibana, but I have had no problems querying Elasticsearch directly. It also supports scripted queries if the built-in aggregations aren't enough.
Of course, then you have to build your own visualizations with the results...
Using one of their examples, this:
{
"query": {
"filtered": {
"query": {
"bool": {
"must": [{"match": {"title": "python"}}],
"must_not": [{"match": {"description": "beta"}}]
}
},
"filter": {"term": {"category": "search"}}
}
},
"aggs" : {
"per_tag": {
"terms": {"field": "tags"},
"aggs": {
"max_lines": {"max": {"field": "lines"}}
}
}
}
}
title=python description!=beta | stats max(lines) by tags
s = Search(using=client, index="my-index") \
.filter("term", category="search") \
.query("match", title="python") \
.query(~Q("match", description="beta"))does exactly that with a reasonably good subset of SQL and ES query language mixed in.
It operates in 2 modes; in one it runs the query, in another it spits back out what the equivalent ES JSON query is. We use this as a quick prototyping tool and modify the ES query as needed, as most of us here still "think" in SQL for a lot of things.
I have had some decent luck with sending syslog into mtail and counting generic word events like 'error' and 'warning' as a way to do "something might be wrong" alerting.