Hacker News Reader

Comment by kevindong | Hacker News Reader

The main counterpoint is what do you do when the logging system goes down at the same time there's an incident? Do you then just rely purely on metrics/customer reports to try to debug?

View on HN

When you’re tailing logs on a server there’s still a “logging system“ (your ssh connection, tail -f, a disk with finite space) that can still “fail”. Probably in either case you’d work to fix logging and you’d hope that the system is simple enough to fix.

Logging systems are annoyingly complicated and there are a lot of moving pieces that can fail. Way more than reading a file over SSH.

This kind of access shouldn’t be your go-to but it’s important to have a battleshort.

What's complicated about a syslog server? I suppose some applications like to spew out multiline logs (which are something you just can't process outside the source while maintaining sanity) but seriously, when you actually consider these things while writing the application, you don't need all that much to have something workable.

I was going to make a note that an rsyslog collector would be my only exception to this because of how simple it is but you still have to be careful because by default rsyslog is okay with losing logs in-flight.

But I would be lying if the typical logging setup was this simple. It’s basically all remixes of ELK and while they’re fantastic when they work there are a lot many moving parts that can individually fail.

Reply on news.ycombinator.com