Amon - Python-powered server monitoring, logging, error reporting with JSON API
thechangelog.com
thechangelog.com
The free version is single server (monitors itself) and doesn't have alerting. Monitoring without alerting has its uses, but that's not what I'm looking for. Also, single server isn't what I want. There appears to be a planned payed version that supports multiple servers and alerting (check the page source for a link).
In a previous HN discussion of Amon (http://news.ycombinator.com/item?id=3313991) it was mentioned that the install script had errors. Looking at the install script was the first thing I did. It's kind of insane. It's the kind of thing it might be OK to do on your own servers, but asking others to run it is really dangerous.
The installer starts off by running "type dpkg &> /dev/null". If that exits cleanly is downloads a Debian version of the installer and executes it. If type doesn't exit cleanly it does the same check from rpm and runs the rpm version of the installer. dpkg is available in the standard Fedora repository. rpm is available in the standard Debian repositories. /etc/debian_version and /etc/redhat-release might be a better check.
The installer goes on to download binaries on MongoDB and install them in /usr/local/mongodb. It puts the mongo client in /usr/bin. Does "apt-get install mongodb" not work? It consistently does "tar -zxvf filename.tar.gz > /dev/null". tar -v is verbose, why direct verbose output to /dev/null? The init scripts are the client and server, written in python. It works but that's odd, I want my executables in bin or sbin.
There seems to be a general lack of systems understanding, which is not what I want from a system monitoring tool.
If you are looking for server to accept and store application logs, check out logstash (http://logstash.net/). It will accept pretty much any non-binary input on any protocol, convert it to JSON, and output it to pretty much any storage system using any protocol. E.g., input AMPQ -> output Redis, or input syslog -> output MongoDB. In includes graphing and searching of your logs if you use the default output storage.
For the Mongo installer - I really, really, really wanted to install it with one line of code, instead of writing 100 lines bash spaghetti, but that's not possible - it works on some distributions ( it works fine on Ubuntu 11.04+, but I always had problems with Ubuntu 10.10 and CentOS 5.5 ) and it doesn't work on others.
The problem with the logging are not application logs in general. I started working on that module, not because I didn't like the way I store my log entries, but because - I wanted flexibility on a program level. Every programming language out there has a diverse set of datastructures, but the logging modules are always limited to strings and levels. So you can't really "personalize" and "humanize" your log entries and also - you have one additional step - to convert the log entries to strings. And that depending on the language is between 1 and 4-5 lines of code. And these lines really add up.
I fixed the OS detection script, thanks for that
I'm not sure what the issue is with MongoDB. But if you aren't aware there is the EPEL (Extra Packages for Enterprise Linux) repository for RHEL and CentOS. It's a semi-official and safe repository run by the Fedora Project to add addition packages to EL. MongoDB is in there for EL5 (pretty old) and EL6. Also, easy_install is available for Red Hat and Debian in the 'python-setuptools' package.
I know it's common for people to install things form source on their own systems, but I feel that if you are doing something on someone else's system you need to extra careful. And that means ask permission before installing, use package management, and if you can't use package management suggest ways to resolve the dependencies (easy_install, pip, gem) without actually doing it. See the Homebrew install script as a good example (https://gist.github.com/323731). Another thing is following convention. The way you are using /usr/local is how /opt is supposed to be used. If you use /usr/local the files should be in /usr/local/{bin,etc,lib}. If want to create a package directory and have things like 'bin/' in it, use something like /opt/amon/{bin,etc,sbin,var}. http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html
Jordan Sissel, who created logstash, also created fpm (Effing Package Management) (https://github.com/jordansissel/fpm). It makes it easy to package things. It might help avoid bash script hell.
Create an rpm from the files in /tmp/install:
fpm -s dir -t rpm --name amon --version 0.2.0 --depends mongodb --maintainer "Martin Rusev" -C /tmp/install etc/init.d/amon etc/init.d/amond opt/amon
fpm -s python -t dep tornadoIt's still used for network monitoring since that is what routers and switches talk but most people use something else for everything else.
I've see Ganglia, NRPE, Munin, custom servers and my personal favorite: bash scripts with graphite.
None of these quite match what Amon is trying to do but they all can get you an approximation of it. I'm personally going to have to look more at Amon. It might end up installed shortly.