We built a python daemon that owns a shard of monitors, iterates them, reads recent history, and dispatches alerts when appropriate. The key things I thought were important when it was built were:

- stateless (needs to be able to be arbitrarily restarted at any time)

- fast (we need to be able to send alerts right when a job fails)

- simple (we will run pings thru a series of simple evaluators where test coverage is easy)