Hacker News Reader

Comment by mgkimsal | Hacker News Reader

and... 'fixing' it can cause its own issues.

I've been involved in some systems that routinely spit out dozens of warnings per day. After 'fixing' some of them - maybe getting a system down to a few per week, bug reports started coming in that the system was 'broken'. Because people noticed the boxes and messages they routinely ignored were gone, and this must be a problem. Lots of re-education may need to happen on a large scale to actually 'fix' alarm fatigue across the board.

View on HN

I work on a system with notoriously noisy alarming and messaging logging - my first step is to tell the customer to ignore all of it.

I tell them to do two things to tell them how to check system health.

1. Is it working normally? if so, then nothing is wrong.

2. Walk by the hardware once a week and check for idiot lights, check the system dashboard if one of the idiot lights is on. If no error is found, its a hardware problem (failed supply, drive, fan), if an error is found it's a failed server - either way, call us.

There are a handful of cases not fixed by this, but they're limited to time sync failures - which are critical, but generates no user facing alarms.

This triggered a repressed memory. I was once on an ops team responsible for an application that had been turned over to us written in Java. The application ran in Tomcat, and prior to turning it over to us, the external vendor had "helpfully" built some monitoring and alerting for the application. One of the metrics they monitored was how many errors per minute and per hour were happening the application's logs. When they turned this application over to us, they did not properly hand over the monitoring system or even tell us that it existed, and it was sending alerts to another external vendor team which had no association to my ops team.

Several years went by... then because we finally had the time our ops team decided to audit and improve this application, including rewriting significant portions of its code as was pretty typical when we took over an application built by an external vendor. Long story short, we resolved almost all of the errors by the expediency of simply correctly interacting with the database behind the application instead of relying on whatever horrific dumpster fire output to the DB that Hibernate was doing. That's when things got super interesting, because the external vendor team that was receiving these monitoring alerts (unbeknownst to my team) had more than 100% turnover in the intervening period of time, having lost all knowledge about this, but suddenly starting getting a bunch of critical urgency alerts going off because the application was no longer emitting the amount of errors expected for the alarm thresholds. This ended up triggering a massive company-wide incident call, even though there was no external customer impact. I'm leaving a lot of details out, but let's just say it sucked up days of my life and was a massive waste of time for all of that.

Fun example of fixing something and causing an "issue" because behavior changed from expectations/baseline.

"The errors let you know it's working; it's when the errors stop you have to worry."

For old cars we say say "If there ain't no oil under it, then there ain't no oil in it."

Reply on news.ycombinator.com