Hacker News Reader

Comment by com | Hacker News Reader

The advice about logging and metrics was good.

I had been nodding away about state and push/pull, but this section grabbed my attention, since I’ve never seen it do clearly articulated before.

View on HN

Yes. Everyone should spent the small amount of time getting some logging/metrics going. It's like tests, getting from 0-1 test is psychologically hard in a org but 1-1000 then becomes "how did I live without this". Grafana has a decent free tier or you can self host.

The logging part is spot on. It has happened so many times when I thought, "Oh, I wish I had logged this.", and then you face an issue or even an incident and introduce these logs anyways.

It is a balance. Too many logs cost money and slow down log searches both for the search and the human seeing 100 things on the same trace.

Yeah, absolutely. But the author's idea of logging all major business logic decisions (that users might question later) sounds reasonable.

Yes. I like the idea of assertions too. Log when an assertion fails. Then get notified to investigate.

The trick here is to log aggressively and then filter aggressively. Logs only get costly if you keep them endlessly. Receiving them isn't that expensive. And keeping them for a short while won't break the bank either. But having logs pile up by the tens of GB every day gets costly pretty quickly. Having aggressive filtering means you don't have that problem. And when you need the logs, temporarily changing the filters is a lot easier than adding a lot of ad hoc logging back into the system and deploying that.

Same with metrics. Mostly they don't matter. But when they do, it's nice if it's there.

Basically, logging is the easy and cheap part of observability, it's the ability to filter and search that makes it useful. A lot of systems get that wrong.

Nice. I'm going to read up more about filtering.

Reply on news.ycombinator.com