The Lost Art of System Administration
matt-rickard.com
matt-rickard.com
The technical stuff changes but Systems Administration remains the same practice. You figure out how to put together a complex system, how to maintain it, keep it highly available. Operations is the work of keeping the business afloat through IT, and Systems Administration will be a part of that as long as there are Systems to Administrate. The title will change, because apparently people are squeamish about being lumped into a group with people who aren't all cutting edge. But the art won't be lost. It lives in every woman who stays up til 3am to ensure maintenance goes as planned, and every man who tells the developers, "No, I'm not giving you root on all the servers, tell me what you want to do. You just want to copy a file? You don't need root for that. ...."
I'm a developer, so I'm very biased.
My interactions with system administrators were more along the lines of "I can't give you access to this system, you have to [come to my desk / hop on a zoom call] and tell me what to type into the shell." and me "Please type tail dash f, no, not slash, dash, the horizontal line thing" and a whole lot of back and forth like this, after which I thank them politely and they feel very accomplished while I'm left guessing why that entire department is necessary...
But also, are you shipping all the logs? Of course you're shipping application logs somewhere searchable, what about gc logs, syslog, kernel logs, etc? Is it never necessary to run ad hoc commands on your production systems?
> "Is it never necessary to run ad hoc commands on your production systems?"
Ad-hoc commands in production AS ROOT should be exception, not norm, and reserved for on-call members, not every developer working on team.
Even the most competent people, sysadmin or developer, can make a mistake.
I did not say no commands should ever be run on production, there could be non-root read-only accounts. I am opposed to everyone having root on production. I have seen enough data loss incidents or full blown outages because of mistakes.
SaaS DB like RDS is another example. I've seen mysterious growth on a read replica, which was fixed by restarting it. Temporary intrinsic tables were not the cause, if you were wondering. No noted errors were logged in its error log it shipped out, and of course the general log wasn't enabled because it generally gets absurdly large. Had this been a self-hosted installation, there may well have been clues that would have been available only by logging into the host.
"Cattle, not pets" is an admirable goal, but sometimes you have to play veterinarian on a few of your herd to figure out what illness is spreading, and how to inoculate against it.
Ironically, these examples are only proving the point made in first comment.
When it comes to debugging running systems, nobody has it better than doctors. They knock the "system administrator" out and go right in with root access via scalpel to do what they know needs to be done.
SysAdmins can be a huge help in devops culture, but sometimes things go wrong and the surgeon needs to get in there.
1. They could give you root access. But then you might start making other changes, and over time that makes everything unmaintainable. By them just typing in what you want this one time, they limit the scope of your access to just one change, they are aware of what changed, and can push back on it. It's basically a crappy change management process.
2. This person has a lot on their plate (a way larger backlog than the average dev) and doesn't have time to research how to do the thing you want them to do. They are asked to be experts at everything, but they can't be. So they ask you to show them exactly what to do. Gets rid of ambiguity, lets you both troubleshoot any unexpected problems in real time.
3. A lot of roles where one person/group has "power" over another, often takes it for granted and unnecessarily gate-keeps and adds red tape and blockers. DevOps is supposed to flip this situation on its head by giving you, the dev, the tools needed to do your job, without you making changes to the system that would affect system stability. This keeps the system stable while giving you agency. Unfortunately, doing all that is much more difficult than most people realize, so it rarely happens.
4. Like every role, sometimes there's idiots. I've met my fair share of both idiot devs and idiot sysadmins.
2. Yes, I'm sure this was the case.
3. Indeed!
4. I haven't! My original comment just took some situation out of context. I know nothing about what else the sysadmins were doing, they probably had other things on their plate about which I knew nothing...
I have however worked with a narcissistic power hungry incompetent sys admin in the past. That doesn’t mean though that I automatically assume that all sys admins are the same. The one I am working with right now is highly competent and a pleasure to work with. And he is smart enough to simply give VM’s with root access to developers who ask for it.
While a lot of jobs can abstract away these things, they are still there, and very real. If more people had an understanding of the underlying operating systems and file systems we could mitigate a lot of vulnerabilities, or find performance issues that may otherwise be obstructed, or a myriad of other things.
Hacker News is in a bit of a bubble in that the answer to everything seems to be "kubernetes" or "$newHotSolution". I think part of it is that a lot of developers haven't actually worked with machines. At one point in time there was a post here about how hard it is to set up a LAMP stack, a task you could hand a first year sysadmin and they should be able to figure it out. Abstraction and automation are nice, but the underlying concepts are still important.
They’re solving two different problems.
Seriously: I run clusters from a few dozen nodes (down from a few hundred at my peak, sigh) down to a trio of Raspberry Pis in my living room. They're overkill by a little bit, but not by much. And it's definitely my ambition to make the tooling even easier and even more powerful, such that every small home can run something with enterprise level reliability.
K8 is where you don't own the kitchen, and you lease it when you wish to cook. You aren't aware of how to maintain the kitchen, or buy the ingredients you choose to cook with. In fact, you aren't even able to tell if a mango is ripe or not when shopping, because you don't shop and don't know how to.
That's what K8s are.
Meanwhile, SysAdmins know how to maintain, manage, run the kitchen... as well as cooking the meal. SysAdmin knowledge scope is greater than K8 knowledge scope for this reason.
What AWS, what docker, what K8s have done, is outsource specific realms of knowledge and skills, so people don't have to "deal with that". But if one is outsourcing knowledge and skills, one cannot claim that this makes the work more sophisticated.
Celery spawns n workers, defaulting to the number of logical processors available. As anyone familiar with cgroups can tell you, this is fraught with problems when containerized, since nearly all mechanisms to detect processor count (or available memory) lead back to `/proc`, which will dutifully report the host's information out to the container. This leads to questions like, "I requested 4 vCPUs; why do I have 4+n threads?"
ORM in general. The worst example I've seen was querying a massive table, and to get n results from the end, was using OFFSET + LIMIT. It was also deliberately not using the PKEY, leading to a full table scan every time the query ran. If you aren't familiar with DBs, it may seem perfectly reasonable that querying a ~100 million row DB would take a long time, when in fact it could and should be extremely fast with a properly written query.
the only real difference is the sense of smugness when it works. In the old days deploying LAMP was a sense of achievement. save for patching, there wasn't much more work to do.
Kubernetes is basically the same level of effort, but the upkeep is a bit more.
Also the networking is batshit, and so is the aversion to swap.
It would be nuts to let every developer tinker with the production server, or the source-code repo, or the fileserver. The private VM gives them a playground where they can learn to play at sysadmin, if they want; real sysadmin, I would contend, is taking the burden of responsibility when shit happens. Nobody cares much about wizards, until shit happens; the wizard then becomes an essential scapegoat. The newsgroup alt.sysadmin.recovery wasn't created for nothing.
I think that it might be a good idea to have most of the configuration for servers be based on Git repos, with something like Ansible or another such solution (Chef, Puppet, Salt), so that server configuration wouldn't be treated that differently from versioned code (which also serves as a historical record).
Don't give developers access to just push changes for production servers, but absolutely let them create merge/pull requests with changes to be applied once proper review is done: ideally with a description of what these changes accomplish, references to other merge/pull requests for development/test/staging environments where they were tested beforehand (perhaps after prior tests against local VMs) and a link back to whatever issue management system is used.
Then, have an escape hatch that can be used by whoever is actually responsible for keeping the servers up and running, in cases an update goes bad or other manual changes are absolutely necessary, but other than that generally disallow anyone to access the server directly for whatever reason (or at least remove any kinds of write permissions).
Personally, I'd also argue that containers should be used to have a clear separation between infrastructure and business applications, but that's mostly the approach to use when dealing with web development or similar domains. Regardless, I find that it's one of the more sane approaches nowadays, the Ops people can primarily worry about the servers, updates, security, whereas the developers can worry about the applications and updates/security for those.
I feel sad for my kids who will grow up with iPads and Windows 14 PCs with not the slightest clue on how things are running and the OS shielding them from crashing the thing or running viruses for fun.
Same things for young lads that come out of university today with their degrees. They're well spoken and can recite algo theory like nothing but still have no clue what's underneath and why it works.
Have we reached that level of the civilisation that goes extinct, for forgetting how its own inner machines work?
If you’re looking for buttons, something to hack on and break, try: https://www.raspberrypi.com/products/raspberry-pi-4-model-b/
Personally I think it’s never been a better time to be a hacker. $5 PCBs from free and good CAD software and one can even make their own ASICs now. I wish I was a kid today!
Today the iPhone is one of the biggest status symbols the beautiful people covet, the most important thing in their lives (their Instagram account) depends on it. Yet none of them have a clue how it works. Like the Morlocks, hackers kind of understand how it works and can maintain it to a degree but can't really build a new one. We may be only one supply chain collapse and food shortage away from HG Wells' fiction becoming reality... :)
Probably. Almost nobody under the age of 40 works in the "deep weeds" anymore (firmware, kernel, casually "Ring 0 and below" spaces). A decade ago, it was about 30. I'm the last wave, it seems, of people in these spaces, and that's quite terrifying.
But I get it. The hardware is maddeningly complex, rapidly getting more complex to solve the problems created by the last complexity, and the whole stack is cracking and crumbling around the edges (the uarch vulns and Intel pulling SGX from consumer chips are a reflection of this). Another decade, this entire low level group will have retired to something far away from computers, and... we'll see what happens. Nothing good, I fear.
On the hardware side, it was very easy to learn how to fix said machines, with schematics provided in the back of the user manual. This level of understanding is not as easily possible today, and my understanding grew outward from that.
I think two things, the need to understand low level to work with hardware competently in the past, and that it was fun to work with hardware in the past, have led to a theoretical golden age of people who understood it all.
I have deep understanding of baremetal, right through to frontend. Part of that comes from working with computing as it changed and morphed.
People today are graduating, entering the field, but are starting at a higher level.
One other thing to bear in mind. When I entered this field, it wasn't to make money. Salaries were abysmal, and on top of that, the field was seen as "boring" and "stodgy" to a great many. And there were often even poor employment prospects!
This meant that people who worked in computing prior to 2000 or so, did so because of joy. They didn't think "Oh, I can make a good living here, I choose this job". In fact, people would tell you not to work in computing, who on Earth would hire you!
So people from this era often were in this field, because they were drawn to it not for income, or job security, but instead because it was more important to work with the fun and joy of computing.
While there is absolutely nothing wrong with entering a field to make a living, or for job security, the motivations are different.
An example here ; using a computer is different than building computing. Enjoying using a web browser, enjoying interacting online, that is not "enjoying computing". Yet most people I speak of, were building circuit boards, coding in basic, when kids.
How many CS graduates were coding apps, building webpages, for 100% fun, without any encouragement when 5?
So I think the same capability is there, it is just that most people today are not interested in computing. They just happen to make a living there, and so those enjoying computing as a primary are awash in a sea of those who couldn't really care less.
Its creator, Eben Upton was a director of studies at Cambridge university and realised that there were fewer CS applicants than in the past and that many of those that did apply knew far much less about computers and programming through there own "tinkering" than previously.
He saw that this was due to modern computers being far more opaque than the systems he had used as a kid (like the BBC Micro). So he decided to build something to address that issue that was open enough to understand the workings and cheap enough that people wouldn't be discouraged from tinkering due to the risk of breaking it.
It seems to have worked quite well...
But with the global availability and affordability of computing devices plus the global knowledge base of the web curious geeks never had so much opportunities to learn and practice.
Here is just a link which i hope will bring some joy to your day : https://steamcommunity.com/sharedfiles/filedetails/?id=28422...
Other sources of hope are retro computing communities, emulation devs, FPGA scene (Mister) for hardware side, assembly devs / optimization communities for software side.
Hardware changes slowly make current system design choices obsolete. Persistent RAM, TBs of RAM will require low level OS redesign sooner than expected.
On PCs, maybe, but "kids these days" use microcontrollers like I used meccano parts.
A lot of the best people I know in that field (low level stuff) are younger than me.
I feel sad for my kids who will grow up with ms-dos and high-level languages and not have a clue how things really work.
Whither civilization?
extending this logic. modt people in the wedt can't go food, connect plumbing or fix a broken chair. only 10% of the people will find correct ancor for drywall or fix a flat on a bycicle
This thought was expressed in Isaac Asimov's short story "The Last Question" [PDF]: https://physics.princeton.edu/ph115/LQ.pdf
> Alexander Adell and Bertram Lupov were two of the faithful attendants of Multivac. As well as any human beings could, they knew what lay behind the cold, clicking, flashing face -- miles and miles of face -- of that giant computer. They had at least a vague notion of the general plan of relays and circuits that had long since grown past the point where any single human could possibly have a firm grasp of the whole. Multivac was self-adjusting and self-correcting. It had to be, for nothing human could adjust and correct it quickly enough or even adequately enough. So Adell and Lupov attended the monstrous giant only lightly and superficially, yet as well as any men could. They fed it data, adjusted questions to its needs and translated the answers that were issued. Certainly they, and all others like them, were fully entitled to share in the glory that was Multivac's
I wonder whether they'll have to deal with CrashLoopBackoff errors in 2061.
I do very much agree that use of command line is declining, though; in general, technology - both its physical form and digital interfaces - has become more closed to the user. Really hope that changes.
These days it's devops, but now there are no specific devops people and it's yet another task unrelated to engineering regularly thrown onto engineering's plate.
They had an IT department. Things were smooth. Nothing ever went bad.
So they sacked 50% of them thinking they do nothing but hang around... and everything went to hell.
Someone can probably point to or correlate that story better.
But essentially the human nature of business is, if something runs smooth, it needs fixing.
Anyone have a link and/or know more?
For example, in place where I work (as developer) operations found good combination of kernel, driver version, NIC chip revision and sysctls, which allows us to use 20% less hardware for our typical load (or allow us to serve 20% more clients on existing hardware without buying new one), and it is substantial for us.
Yes, I, as developer, don't need to be system administrator, for me it is containers and CI and automatic deployment, infra as a code, and all this stuff.
But somebody NEEDS to build & support this infra. And change faulted SSDs.
We used to think “the year of the Linux desktop” would be the extension of the Linux ecosystem to include both servers and desktops. Now that Linux personal computers are finally mainstream — at least for people like us — will it instead turn out to be an exodus from the former to the latter?
This is a complaint I used to hear about young software developers even a few decades ago. It's the same today, but those accused then are now complaining the same for the current crop of developers. I guess it is a skill that people pick up late and slowly, but they eventually do it nonetheless.
Honing that skills can be slow.
Everything is much more stable than it used to be. It might seem not to be the case, but I assure you, it is. Once something works, it tends to keep working.
I've been a Linux sysadmin from the late 90s through to the mid/late 2010s, now I'm doing DevOps. My sysadmin-spidey-sense has helped me to solve so many issues, and my foundation of knowledge in systems/networking/etc helps me design better code and infrastructure.
I switched from Arch to Ubuntu because they shipped a broken avr-gcc despite upstream GCC saying "roll it all back, guys", because it was the newest version. Doesn't matter that it produced deeply faulty output, newest is best!
"The network is running perfectly, why are we paying the sys admins?"
Valves have been replaced by transistors, transistors have been replaced by integrated circuits.
In fact, I'd argue that kubernetes was really made to remove sysadmin from being needed. For large kubes obviously some skill is needed but in the average case a developer can do 99.9% of all the work needed. It was a huge boon for businesses that would otherwise have to pay both developers and sysadmins. Now they can pay one of them half the value!
The majority of other sysadmin tasks have been offloaded to cloud providers. For example, AWS makes setting up a network topology/compute/etc very easy. Sometimes these tasks are handled by SRE, a lot of times they are handled again by developers. Terraform really brought everything together in an absolutely brainless mechanism to get things started (with the subsequent pitfalls).
They’re paying the other half of the value to the cloud provider, more or less. Ideally, perhaps a bit less. But that’s how business works. The cloud provider is operating these systems much better than 99+% of companies could hope to do.
k8s is a great lever for sysadmin. It's the kind of tool we spoke of in fevered tones when we shared legends of "automating our job so we would only need to pick they paycheck". Does it mean there's less need for absolute Ops-side numbers? Sure. Did we have the numbers before? hahaha, fat chance, nope. So in my experience, moving onto k8s often means my job involves less understaffing now, less panic, and less cursing. Most importantly, less reasons to visit Scary Devil Monastery (alt.sysadmin.recovery).
And if the low amount of work starts to get depressing... you can always take another job[1] or start engineering the shit out of the infrastructure. Bonus points if you make it without making it problem for future you (so documentation and the like) and without raising costs.
[1] A polish joke about sysadmins: "Hey, Joe lost a job, you heard?" "How can he deal with having only 5 full time jobs?"
> The majority of other sysadmin tasks have been offloaded to cloud providers
Yes. There is a word for this process: deskilling — https://en.wikipedia.org/wiki/Deskilling
It is typically followed or accompanied by a loss of autonomy and/or other privileges in the workplace
I've yet to meet a dev who knows how or wants to create and maintain the IaC to manage the infra.
> Terraform really brought everything together in an absolutely brainless mechanism
Understanding Terraform's limitations and being able to work around it in a way that is both readable and scalable is most definitely not brainless. Sure, I can write some .tf files to spin up an EKS cluster pretty easily. Load balancer? I can also do that in Terraform, or I can let the cluster manage its own. Uh-oh, a fork. Which one is better? What if I want to have multiple AWS accounts; how do I properly delegate IAM permissions across them? Etc.
And you should too, BTW. Or one day you learn, that cloud you choose to use don't have BOTH backups and redundancy, for example. Learn in a bad way.
Or, maybe, you'll learn, that you need 10x more performance now (more throughput, less latency), and this cloud can not provide it at any cost.
I was just moving an EBS volume between regions and doing the partitioning and thinking "wow, you just don't do this much anymore".
Imagine after buying a macbook that you also need to hire a sysadmin to keep it running smoothly? The idea is laughable.
> Learning how to effectively use the command line. Unix philosophy. Pipes, scripting, and tooling. So much of programming is stitching things together.
This is really bad. Much of the complexity in contemporary software architecture arises from this mentality.
"Do things by stitching programs together" only works for relatively simple tasks. For everything else you need to program the thing properly.
One of the worst ways this manifests is in how websites are built: use a combination of a scripting language, a database server, an http server, a memory cache server, etc.
I'm really grateful we have things like Go and SQLite that make it possible to create fully functional websites as self-contained programs instead of the hot mess that everything else uses.
Someone who is running a newer Linux desktop distro is not going to need sysadmin experience to web browse and read email. The same way someone using Windows or a Mac for a desktop distro isn't going to need sysadmin experience, either.
As for your websites, no one is going to build a simple website that needs a dedicated database server, a "memory cache server", or whatever. Those issues become relevant with scale.
Do you buy and string together several MacBooks to run your business's custom built ERP system 24/7/365?
There isn't a vast ecosystem of open source solutions to enterprise problems because that's not what macOS is, or attempts to be.
I can't tell if this is a serious response or not, this comparison seems pretty ridiculous.
What do you think functions are?
> I'm really grateful we have things like Go and SQLite that make it possible to create fully functional websites as self-contained programs instead of the hot mess that everything else uses.
Yes, because as someone who will be called to troubleshoot it, I definitely want to trudge through whatever logging you may or may not have implemented in this monolith to solve a cache miss or 5xx error. /s