Graph everything with Graphite

FWIW I've found influxdb considerably easier to install and manage than graphite (graphite doesn't play well with virtualenv, which makes dependency management horrible, compared to influxdb's single static binary)

Also, I can see logging dictionaries being much more efficient and useful than logging single values -- with graphite if you want to track page hits per section of your site (of which you have 10) per user (100) per browser (5), you end up with 5000 individual metrics, and you need to have thought of them in advance. With influxdb you can log {"section": "front page", "user": "bob", "browser": "firefox", "hits": 1} as a single metric and then use an SQL-like query to filter by section / user / browser (or any combination of those) as and when you want to.

TBH the only thing I miss going from graphite to influxdb+grafana is the tree of metrics (grafana has autocomplete once you start typing, but you can't just browse) and a few of the rendering functions (moving average).

Yep. I'll repost an older comment of mine:

Development of graphite is effectively dead. The datastore component (carbon and whisper) has design issues, and the official replacement (ceres) hasn't seen any commits this year. There are a some alternatives, though.

For data storage Cyanite [0] speaks the graphite protocol and stores the data in Cassandra. Alternately, InfluxDB [1] speaks the graphite protocol and stores the data in itself

To get the data back out, there's graphite-api [2] which can be hooked up to cyanite [3] or influxdb [4]. You can then connect any graphite dashboard you like, such as grafana [5], to it.

[0] https://github.com/pyr/cyanite [1] http://influxdb.com [2] https://github.com/brutasse/graphite-api [3] https://github.com/brutasse/graphite-cyanite [4] https://github.com/vimeo/graphite-influxdb [5] http://grafana.org

How does Graphite, and Influxdb compare to say an ELK stack?

Right now I'm in the process of setting up a Log aggregation/metrics system for PCI logging, and general server stats. Right now I'm using Elasticsearch, with Logstash and Kibana, and then piping OSSEC logs into Elasticsearch for security logging. I like the idea of this ELK stack, but it's not very polished imo.

Only way is to make it self documenting, when a metric is set up, when you are navigating the graphs etc there should be some text defining where this data came from. It also depends if lack of data means something to your system. In my system a lack of data for a metric for a time span x implies that something is wrong and should be alerted. In some systems this might not matter. This provides a self fulfilling habit of removing dead metrics

Obviously, you need metrics to monitor the metrics.

But more seriously, I've attacked this kind of problem in the past by agreeing on a common set of first-look metrics. What is a one page set of data you start with when trying to identify a problem in a given system/subsystem.

After than, other metrics must exist to solve or monitor a specific issue, or be in reusable shared sub-reports with some stated overview. After than, you can think about automatic deletion after a specific timeframe if they're in sandboxes and not connected to some documented useful purpose.

We are working on building a system that combines events and metrics (www.jut.io), and one of the things we've been thinking about is how to store metadata about your metrics. It's not only stuff like min/max values, but almost anything you could think of - developer, feature name, expected frequency, etc. Would a concept like this be useful?

I ended up rolling my own replacement. My biggest problem with Graphite was that it managed to grind an expensive large RAID array into the ground with a relatively small number (in my eyes) of metrics. We had the realisation that we'd waste a tremendous amount of hardware or have to cut down drastically on our data collection if we were to roll out Graphite across the board.

(And yes, we had crashes too)

The reason for the disk grinding was simple: The whisper storage system is ridiculously inefficient as it does tiny writes all over the places, and an excessive number of system calls to boot.

In our case, I decided we don't care if we lose some data if a metric server crashes - if it becomes an issue we'll run two or more vms on separate hardware and feed half our samples into each -, so the first step was to write a simple statsd replacement that shovels 10 second intervals of data into Redis with disk snapshots turned off, coupled with a small daemon that rolls up (I've hardcoded roll-up intervals as it made it easy to use naming of the keys to use "keys <timestamp for start of each interval to roll up>-<postfix for type of period e.g. we use 10 second then 5 minutes, then hourly>-*" to retrieve the keys of the objects to process each step).

We could've easily beat Carbon/Graphite on the same system just by doing more efficient disk writes, but since we were first going to replace it I figured I might as well keep things in memory.

Then a tiny replacement for the subset of the Graphite HTTP API we used for our graphing (if we'd relied on Graphite itself for our dashboards I'd have thought twice about this...).

Lastly a tiny process that archives a final roll-up of data past 48 hours (currently) to CouchDB for if/when we need to do longer term historical trending.

I keep wanting to talk to our commercial director about letting me release some of this code, though a lot of it is probably too specific to our needs to be all that useful to others (e.g. as mentioned, we only support a tiny subset of the functionality of the Graphite HTTP API, as I've only cared about being able to do the averaging and filtering etc. that we actually use). In general, though, if you don't use Graphite for the actual dashboard, replacing it is surprisingly little work.

We've had the same problems. Unfortunately we have so many metrics being stored in Graphite that we've come to rely on it for different monitoring tools. This post has some great information on how to scale out Graphite to hopefully make it more stable and efficient for you: https://grey-boundary.io/the-architecture-of-clustering-grap...

Grafana is surely the way to go when composing dashboards for your Graphite data.

Be sure to check out the screencasts to get a nice overview and quickstart of the features: http://grafana.org/docs/screencasts/

Also if you are no longer using the Graphite web frontend consider switching to graphite-api: http://graphite-api.readthedocs.org/en/latest/

Could you give us a few examples of a better name?

That's kind of how a lot of new products are named, in and out of tech. It can be difficult to create a new word that's agreeable in the way it sounds, and preferably carries a little meaning tied to the product.

The search engine issue is a very real problem, I agree. Like in the early days of the Go programming language. Eventually the indexes got better, especially if you gave a little more context to the search (ie. "http client libraries for go"). And, of course, learning early on that it was going to be a known problem so I started using golang in my searches.

It is quite difficult to come up with a good catchy name that has no meaning in the real world, and yet is easy to memorize.

Having said that, I agree with you (and don't understand why people were down-voting, you make a perfectly valid point?). I had real trouble a while back searching how to use "chef knife" and how to write "chef recipes".

I've used collectd's tail module for this. See https://collectd.org/wiki/index.php/Tail for details.

Ryan Smith's "Building Metrics From Log Data" is an interesting talk about this:

http://vimeo.com/68183624

Also check out Heroku's Lumbermill project, which handles extracting router metrics on their platform:

https://github.com/heroku/lumbermill

If you're on AWS, CloudWatch is capable of ingesting log data and extracting metrics via pattern-matching:

http://docs.aws.amazon.com/AmazonCloudWatch/latest/Developer...

When I used Graphite with Nagios, I bypassed all the Nagios data-collection and graphing features. Instead, I funneled all the data into Graphite and used check-graphite to alert on it:

https://github.com/pyr/check-graphite

Thanks for this link. Most of the data I deal with is in relational SQL databases so this looks very promising for my use cases. Maybe its just me but most of the graphing libraries and APIs are geared towards JSON data.

It should be mentioned that there is a huge bunch of various implementations of the statsd pattern; depending on ones existing infrastructure one may prefer one or the other. Heres a comprehensive list of them: http://www.joemiller.me/2011/09/21/list-of-statsd-server-imp...