Comment by throwaway892238 | Hacker News Reader

Comment by throwaway892238 | Hacker News Reader

throwaway892238

Most of those things listed are just technical details that change over time. Before all the Linux trivia there was the Solaris trivia, AIX trivia, Novell, HP/UX, Windows NT, OS/2, VMS, AS/400. Alpha, Sparc, VAX, tape robots, IPX/SPX, Token ring, VT100 dumb terminals. Old heads have forgotten more technical crap than exists today. (I can barely remember how to set up a Cisco router, and I used to work there.. I do, however, remember to turn off spanning tree)

The technical stuff changes but Systems Administration remains the same practice. You figure out how to put together a complex system, how to maintain it, keep it highly available. Operations is the work of keeping the business afloat through IT, and Systems Administration will be a part of that as long as there are Systems to Administrate. The title will change, because apparently people are squeamish about being lumped into a group with people who aren't all cutting edge. But the art won't be lost. It lives in every woman who stays up til 3am to ensure maintenance goes as planned, and every man who tells the developers, "No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. ...."

> No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that.

I'm a developer, so I'm very biased.

My interactions with system administrators were more along the lines of "I can't give you access to this system, you have to [come to my desk / hop on a zoom call] and tell me what to type into the shell." and me "Please type tail dash f, no, not slash, dash, the horizontal line thing" and a whole lot of back and forth like this, after which I thank them politely and they feel very accomplished while I'm left guessing why that entire department is necessary...

In a properly designed and deployed systems, no one (including sysadmin) should have a need to run tail/ less/ grep or whatever.... Logs should be in what ever centralized logging system is in use.. Metrics should be in what ever centralized metrics system is in use.. Config and secrets update should be possible from CI/ CD/ build system, without a full compile + test + deploy pipeline. Need of anyone having access to production to be able to run tail (or anything of that nature) is a sign that system is not well designed, and compensated by... "Give me root access everywhere". Let's not forget user privilege separation came in for some serious reasons.

You're of course right.

But also, are you shipping all the logs? Of course you're shipping application logs somewhere searchable, what about gc logs, syslog, kernel logs, etc? Is it never necessary to run ad hoc commands on your production systems?

How are gc/ syslog/ kernel logs are special and not sent to central logging? Ad-hoc logs (such as thread dumps etc.) can also be triggered via safe application end points from restricted locations.

> "Is it never necessary to run ad hoc commands on your production systems?"

Ad-hoc commands in production AS ROOT should be exception, not norm, and reserved for on-call members, not every developer working on team.

Even the most competent people, sysadmin or developer, can make a mistake.

I did not say no commands should ever be run on production, there could be non-root read-only accounts. I am opposed to everyone having root on production. I have seen enough data loss incidents or full blown outages because of mistakes.

Well done you, I guess? Nothing to disagree with. I'm just amazed this can be actual reality :)

It takes a while to get there - and a lot of discipline of those involved.

In an ideal world, you wouldn‘t need that. But most systems never matured to be considered being in an ideal state / world.

As the other child comment said, are you shipping _all_ logs? For a K8s cluster of a few nodes, sure, you might send `/var/log/*` out, but that doesn't scale. If you have weird Kubernetes problems like mysteriously dying containers that don't trip 137 (hello, non-init processes [0]), you will need to dive into kernel messages. For this particular example, there is to my knowledge one poorly-maintained package [1] that fixes this, but it also requires at the very least some Linux capabilities that might raise eyebrows with security, and as written it requires privileged containers. Thankfully this particular example will hopefully be fixed with cgroupsv2, but still.

SaaS DB like RDS is another example. I've seen mysterious growth on a read replica, which was fixed by restarting it. Temporary intrinsic tables were not the cause, if you were wondering. No noted errors were logged in its error log it shipped out, and of course the general log wasn't enabled because it generally gets absurdly large. Had this been a self-hosted installation, there may well have been clues that would have been available only by logging into the host.

"Cattle, not pets" is an admirable goal, but sometimes you have to play veterinarian on a few of your herd to figure out what illness is spreading, and how to inoculate against it.

[0] https://github.com/kubernetes/kubernetes/issues/50632

[1] https://github.com/transferwise/oomie

Seems "not well deployed system" or "system not well understood by developer", what stops one reducing retention to few hours for node kernel logs or having read only access to certain parts, if at all there are so many kernel/ sys logs. I have never come across a system where application logs were not multiple orders of magnitude more in volume than syslogs. Not sure what RDS vs in-house DB example was meant to convey.

Ironically, these examples are only proving the point made in first comment.

Anecdotally I've had the opposite experience. Majority of developers I've interacted with didn't understand how anything systems admin side worked. This isn't to say all developers are like that, but that you may be more of an outlier in recent times. Also your admin team sounds awful to work with.

A metaphor that just occurred to me.

When it comes to debugging running systems, nobody has it better than doctors. They knock the "system administrator" out and go right in with root access via scalpel to do what they know needs to be done.

SysAdmins can be a huge help in devops culture, but sometimes things go wrong and the surgeon needs to get in there.

Admittedly, that experience sucks. But there's a couple reasons they did that:

1. They could give you root access. But then you might start making other changes, and over time that makes everything unmaintainable. By them just typing in what you want this one time, they limit the scope of your access to just one change, they are aware of what changed, and can push back on it. It's basically a crappy change management process.

2. This person has a lot on their plate (a way larger backlog than the average dev) and doesn't have time to research how to do the thing you want them to do. They are asked to be experts at everything, but they can't be. So they ask you to show them exactly what to do. Gets rid of ambiguity, lets you both troubleshoot any unexpected problems in real time.

3. A lot of roles where one person/group has "power" over another, often takes it for granted and unnecessarily gate-keeps and adds red tape and blockers. DevOps is supposed to flip this situation on its head by giving you, the dev, the tools needed to do your job, without you making changes to the system that would affect system stability. This keeps the system stable while giving you agency. Unfortunately, doing all that is much more difficult than most people realize, so it rarely happens.

4. Like every role, sometimes there's idiots. I've met my fair share of both idiot devs and idiot sysadmins.

1. I don't want root on prod servers, sounds like a nightmare!

2. Yes, I'm sure this was the case.

3. Indeed!

4. I haven't! My original comment just took some situation out of context. I know nothing about what else the sysadmins were doing, they probably had other things on their plate about which I knew nothing...

I like the list of OS you give, but System Administration is not the configuration of a system but instead thousands, with a mind towards performance and allocation of resources, but also includes stressing systems to see what will break. What is being administered is actually people, not machines, and perhaps ironically, relationships become of fundamental importance. Generally, sysadmins are not software developers; they are technology generalists and experts at any operating system and every application, even those they have never used.

I am a developer. Give me a VM I have root access to and walk away. That’s all I need from you.

Right. And then we’ll see how secure your app is running after you follow a bunch of random blog posts telling you to make all files world-writable, or to disable SELinux, or just turn off the firewall. And then the inevitable “it works for me” when we try to push to production and it all breaks.

Just assume for a second that you are talking to an experienced competent developer here. I have never done any of the things you mention. And I have never worked with a developer who did any of the things you mention.

I have however worked with a narcissistic power hungry incompetent sys admin in the past. That doesn’t mean though that I automatically assume that all sys admins are the same. The one I am working with right now is highly competent and a pleasure to work with. And he is smart enough to simply give VM’s with root access to developers who ask for it.

Reply on news.ycombinator.com