You should have Telegraph running on every system you have collecting stats and sending it to InfluxDb with sample rate around 15s and +- 5s of jitter.
Each critical process should have an uptime counter. You should create both Deadman alerts (No contact for 5m) and Uptime alerts (uptime falls below 60s).
s/Telgraf|InfluxDb/YourMonitoringSystemOfChoice/
Bonus: If you're running JVMs, you can expose _all_ of the JVMs stats to Telegraf via Jolokia, which can run as a Java Agent and requires 0 code changes. The JVM has very very detailed stats about it's health available by default, making problems very visible (if you look for them).