Aggregating events in Graphite over time | Hacker News Reader

Aggregating events in Graphite over time

obfuscurity.com

For exactly this reason, Metricfire keeps different aggregate views of the data. Observations is one view. Graphite throws too much away, leading to this sort of, admittedly pretty clever, hacking about with multiple functions.

I don't follow. Graphite doesn't "throw anything away". You define your own retention policy and Graphite follows it. You're free to maintain 1-second resolution for 50 years, if you have the storage capacity for it. But that isn't what this article is about.

The example covers a particular metric use-case for annotation-style details, and how to extract a meaningful aggregate from what is otherwise contextual data. Personally I love that Graphite lets me "hack about" with multiple functions. But I'm a data whore, YMMV. ;-)

Perhaps "throw away" is the wrong phrasing. The aggregates are always averages, no absolute counts. After a rollup, the maximum of some value will no longer be the maximum - it'll be the average of a bunch of maximums. There's no type information with which one can maintain this, and no observation/sum counting. For 'data whore' people like you and I, this should matter. :) (or at least, it does to me)

The article seems to be about trying to get an approximation of observation count data out of Graphite, which doesn't really store that.

It seems to me that you're referring to a) other data outside of the scope of this example and b) the fact that Whisper uses average data for rollups. This is a valid issue and one of the reasons for the new Ceres format.

The metrics I demonstrate in the article do not rollup. If they did I'd completely lose the data. That's why I keep them at 1-sec resolution for their lifetime (generally 1 year). In hindsight it seems your original comment was a general critique of one aspect of Graphite and outside the scope of this particular article.

Reply on news.ycombinator.com