Simple server alerts on a need-to-know basis

The premise is great: you should indeed not rely (only) on complex log and metrics solutions for actionable alerting.

The solution is bad: you should not build a monitoring system yourself. Nagios family of monitoring systems have built a number of concepts from years of experience that you will not cover with DIY. Just a few examples not to make this huge:

- host and service dependencies. When a route goes down you do not want to receive hundreds of alerts for everything behind that route.

- health soft and hard state. A temporary (seconds) bad state should not send an alert. Only if the check fails several times in a row.

- state flaping detection. You do not want tens of alerts for something that becomes healthy and unhealthy several times over a short period.

- escalation: display in dashboard, after a time message L1 on chat, after a time notify L2 with SMS, after a time call over phone L3.

- support custom notification methods: for the above example be able to call by phone. Phone services are orders of magnitude more reliable than mail or http api calls.

- 4 service states: ok, warning, critical, unknown. Takes a lot to explain this but 2 states are not enough. For a networking outage you want things behind the failed point to be marked unknown as they might be ok for a customer taking another route to them than your monitoring.

- support SNMP among other things to be able to monitor network appliances.

I could go on but typing on the phone is not fun and I think this is enough for my point. Learn all things Nagios family of monitoring systems have to offer. Then decide if you can really do better with any other solution (you can, it's just not trivial though).

By all means, use email, but why not use MQTT? I have 3 or 4 small self-hosted home servers, and I set up simple mosquitto_pub bash scripts to relay server messages. The main advantages that I see are speed, low resource usage and the ability of multiple people to subscribe to the message topic concurrently. It also means you don't need an email client; if need be a simple mosquitto_sub is enough.

This is great. I use an almost completely different stack of tools for all this (I’m a sucker for GUIs!) but I totally respect the elegance and simplicity shown here.

More importantly the underlying principles on what to monitor, and when to send alerts, are applicable across all sorts of different monitoring setups.

This has definitely given me a few ideas for updating my monitoring config!

A very practical guide and a nice reminder of the power of some of the simple unix utilities that have been around forever.

One minor correction/change I would suggest, in the crontab example within the postfix section:

    MAILTO=root
    0 0 * * *   root   /usr/local/bin/foobar 2>&1 1>>/var/log/foobar.log

This will make it append to an existing log instead of overwriting it.

I don't really understand the interface check. If the network interface is down... send an email about that? Through which network link?

That's one of the reasons you really want a centralized monitoring system; an offline (link down, system down) system cannot inform you that it's offline, a monitoring system can tell you that.

SysAdmin 201, very nicely done.

I await with bated breath for upcoming ML series to get those most elusive lower severity alert ones.