Ask HN: Good alert hygiene resources?

I'm looking for some ways of thinking about the trade offs in dealing with alerts in shared channels. On an app with very large volume, you end up with many random 500 errors. Sending all of those to shared channels (slack) and people stop paying attention altogether. We end up missing that 1 that was actually actionable and an early sign of a critical bug.

As apps progress the specific errors to filter out shift and create a moving target, meaning the alerts have to be consistently updated and you don't really know it's stale until you miss an important signal.

Any success stories on keeping alerts actionable and useful as time passes? Is it just about prioritization/resourcing a chore and no way around it?

Thanks