Comment by RyanGWU82 | Hacker News Reader

Comment by RyanGWU82 | Hacker News Reader

We have a cluster of about 60 servers, and use Monit and Ganglia to manage it. Monit will alert you when certain problems occur -- for example, if the server load is too high, if a service stopped running, or if can't reach the database. Ganglia records data and graphs that data over time. We use it to see how busy our server are getting, or how our peak resource usage compares to off-peak. Monit is important for noticing a problem; Ganglia's useful for diagnosing that problem.

That said, I'm a huge fan of Cloudkick. Their software is ridiculously easy to install and use, and the web application makes it VERY easy to understand what's going on with your services. It's perfect if you're just monitoring one server -- it's free -- but even worth the money if you've got a whole cluster. If I could do it all over again, I'd definitely go with Cloudkick.

You would have to pay $11,388 per year to use Cloudkick with 60 servers and still lose all data >1 year.

I don't get the logic of paying that much for Cloudkick at all. Monitoring tends to be a couple days work (at most) to setup perfectly and then very little on-going. Hardly worth $949 every month IMHO.

In my experience with running nagios it's never as easy as set and forget to keep your monitoring system effective. Even more so when you're growing or scaling and systems are changing more regularly. To me that $11,388 per year is cheap compared to the costs of having a salaried employee spending more time on a slower to implement solution.

If you don't have a full time sys admin it's also one of those things that is very easy to put off or delay or just not keep on top of.

As for the data loss I can't think of a time I've ever wanted to see resource statistics for over a year ago and down times and outages are kept in a problem tracking system.

One thing I've done in the past to help prevent 'configuration rot' of nagios configs is to hook it into the same files that drove our deployments - when the deployment changes, the nagios config changes with it. Nagios supports template-based configs so once the tests are developed it isn't too hard to write some automation that spits out a config file. This worked pretty well for a site with ~350 servers and devices (switches, APC rack PDU's) with ~3000 tests.

Reply on news.ycombinator.com