Hacker News Reader

Comment by amtamt | Hacker News Reader

In a properly designed and deployed systems, no one (including sysadmin) should have a need to run tail/ less/ grep or whatever.... Logs should be in what ever centralized logging system is in use.. Metrics should be in what ever centralized metrics system is in use.. Config and secrets update should be possible from CI/ CD/ build system, without a full compile + test + deploy pipeline. Need of anyone having access to production to be able to run tail (or anything of that nature) is a sign that system is not well designed, and compensated by... "Give me root access everywhere". Let's not forget user privilege separation came in for some serious reasons.

View on HN

You're of course right.

But also, are you shipping all the logs? Of course you're shipping application logs somewhere searchable, what about gc logs, syslog, kernel logs, etc? Is it never necessary to run ad hoc commands on your production systems?

How are gc/ syslog/ kernel logs are special and not sent to central logging? Ad-hoc logs (such as thread dumps etc.) can also be triggered via safe application end points from restricted locations.

> "Is it never necessary to run ad hoc commands on your production systems?"

Ad-hoc commands in production AS ROOT should be exception, not norm, and reserved for on-call members, not every developer working on team.

Even the most competent people, sysadmin or developer, can make a mistake.

I did not say no commands should ever be run on production, there could be non-root read-only accounts. I am opposed to everyone having root on production. I have seen enough data loss incidents or full blown outages because of mistakes.

Well done you, I guess? Nothing to disagree with. I'm just amazed this can be actual reality :)

It takes a while to get there - and a lot of discipline of those involved.

In an ideal world, you wouldn‘t need that. But most systems never matured to be considered being in an ideal state / world.

As the other child comment said, are you shipping _all_ logs? For a K8s cluster of a few nodes, sure, you might send `/var/log/*` out, but that doesn't scale. If you have weird Kubernetes problems like mysteriously dying containers that don't trip 137 (hello, non-init processes [0]), you will need to dive into kernel messages. For this particular example, there is to my knowledge one poorly-maintained package [1] that fixes this, but it also requires at the very least some Linux capabilities that might raise eyebrows with security, and as written it requires privileged containers. Thankfully this particular example will hopefully be fixed with cgroupsv2, but still.

SaaS DB like RDS is another example. I've seen mysterious growth on a read replica, which was fixed by restarting it. Temporary intrinsic tables were not the cause, if you were wondering. No noted errors were logged in its error log it shipped out, and of course the general log wasn't enabled because it generally gets absurdly large. Had this been a self-hosted installation, there may well have been clues that would have been available only by logging into the host.

"Cattle, not pets" is an admirable goal, but sometimes you have to play veterinarian on a few of your herd to figure out what illness is spreading, and how to inoculate against it.

[0] https://github.com/kubernetes/kubernetes/issues/50632

[1] https://github.com/transferwise/oomie

Seems "not well deployed system" or "system not well understood by developer", what stops one reducing retention to few hours for node kernel logs or having read only access to certain parts, if at all there are so many kernel/ sys logs. I have never come across a system where application logs were not multiple orders of magnitude more in volume than syslogs. Not sure what RDS vs in-house DB example was meant to convey.

Ironically, these examples are only proving the point made in first comment.

Reply on news.ycombinator.com