System Admins R.I.P.?
blogs.mulesoft.org
blogs.mulesoft.org
On the one hand they're insulting. The author is clearly wholesale ignorant of the vast majority of sysadmin roles and responsibilities (and why companies need them) and yet feels like they know what they're talking about enough to declare the role dead.
On the other hand, developers with no clue whats entailed in sysadminning are the number one source of our job security, so its a backhandedly good thing.
Please, by all means host your startup's database in a "aaS" solution with ~500 IOPS (except sometimes! at random!) across a 50 - 100ms link. It will make me look like a wizard when I clean up your rookie moves.
A sysadmin would have figured it out in 5 minutes, and lazily dropped by my desk to show me where I went wrong.
In my experience, sysadmins aren't just people who keep things running during the course of Business As Usual, you guys are also a terrifying repository of arcane knowledge.
* Those working for infrastructure providers (IBM, Amazon, Rackspace) focus on the bottom half of the stack: everything from the datacenter's floor plan to switching and VM allocation.
* Those working for infrastructure consumers (everybody else) focus on the upper half: the business's software stack, how to glue it together, and where to run it.
Conclusion: system admin, as a profession, is gradually disappearing. But it's being replaced by 2 better professions: more challenging work, higher perceived value within the organization, and higher demand.
Not many years ago people got paid to, among other things, pick out the appropriate operating system for any given application. Clearly there's not much of a future in that.
These days that same person is responsible for vetting cloud hosting providers, email service providers, and has to be capable of comparing the "old" in-house solutions and bring things inward if there's benefit.
The technology changes, but the role more or less stays the same. The system admin is generally a bridge between business operations and the technology stack, and there's always going to be glue there, particularly for businesses of at least 10 people.
Disclosure: System Admin / Engineer by trade. Weigh my opinions appropriately.
1. I think there's a distinction between Sys Admin and a more management position like Director of IT or CIO. In some companies there's only one person who fills both roles but they are two distinct roles and I think this article refers to the actual System Administration role rather than the management role that selects software and vendors.
2. While I don't think the System Administrator role will disappear completely I do think the focus will switch from one that is more hardware driven to one that's more software/programming driven. I think businesses will expect System Administrators to handle integrating their cloud based services into a comprehensive whole in the future rather than just making sure there's a network to carry data. That's important because it will mean a dramatic shift in the required skill set.
That's a lot of words to describe work that isn't a lot of my time annually. Especially on a team where the load is distributed. Take into account that a few hundred "servers" are virtual machines running on tens of physical servers and there you go.
My skill set already requires me to manage multiple operating systems and applications. My biggest value to the company is problem solving, whether it's engineering a solution to meet new business requirements to diagnosing and fixing a problem that can happen anywhere from the hardware to application layer. Sysadmins don't just sit and look at blinky lights all day waiting for one to go orange.
But what cloud services allow us to do these days is scale services without the friction of requisitioning new hardware that used to happen in the past.
Of course, your sysadmin needs to be familiar with tools to help them manage large numbers of servers before this really begins to matter.
This article presents much the same scenario, but updated for 2011. And make no mistake, it will probably work for small, well managed, web-based companies, full of people that already know plenty about computers and technology. For pretty much every other type of major company though, there needs to be at least one person who's responsibility is making sure that the technology in the workplace can be worked with and not just worked around, and who can be responsible for dealing with the new technologies that come around, and until the cloud can provide such a service, the sysadmin will be alive and well.
They'll never fail, get obsolete, lose our data, get bought out by someone who ruins them, have availability problems, or lock us into using just their service, right? Guys?
OK, dueling strawmen out of the way, let me try to add some value to this discussion. Just like the lower rungs of the value chain are getting eaten up in programming, the lower runs of the value chain in sysadminning are also getting eaten away. Back when I was in middle school I was, I kid you not, routinely told that the comparables to my labor at HTML editing all charged $100+ an hour for making websites in Notepad. (Complete with under construction signs.)
Thankfully, the state of the art has improved such that there are a variety of options for e.g. a church school to have a web page listing upcoming Christmas pageants without having to pay $10k a month on an ongoing basis for maintenance.
You know the old saw about competent sysadmen trying to write themselves out of jobs? Well, don't look now, but in a lot of areas they are winning. My old day job called me a Systems Engineer, which includes basic sysadminning, and over my very short career standard deployment practices for e.g. Rails apps have improved radically in a direction which requires me to spend far, far less work at this aspect of m job. There are "real companies" on similar stacks who have server-to-admin ratios which just 10 years would have been unthinkable. (100 to 1? No problem. It's really just 3-to-1 with heavy automation in play, anyhow.)
Sysadmen in the audience worried about their career prospects should probably try pushing themselves up the value chain. There was once lots of value in babysitting cranky programs which blew up frequently in unpredictable ways. That looks like it is changing. Look at Exchange versus I-can't-believe-its-not-Gmail from the perspective of a firm with less than a thousand employees: heads you have someone whose full time job is to fight outages, tails you do not. Don't be the John Henry in that scenario: remember, he dies at the end of the story.
When I preach this, I modify it to "always eliminate your role". Your job won't go anywhere, but what you're doing for that job should always be changing.
To give one example. We've recently started moving our users to Box.Net because the hassle of having a VPN connection for each user just to have them connect to a server with ShadowCopy was becoming too much. Box.Net requires no such VPN connection and its syncing is superior to ShadowCopy bandwidth wise.
That said, there's a screaming business opportunity for a group of enterprising sysadmins who want to set up HIPAA-compliant PaaS offerings for medical information shops, or something similar for banks that won't make the SEC or FDIC freak out.
Too much existing infrastructure. Too many systems that aren't web-UI apps with cloud backends.
Maybe in ten years, this will be more plausible.
But who knows what will happen in the future...
When Toyota retools a factory, they don't have robots build/deploy the robots, there are engineers that "re-tool". This is exactly what is happening at Google/Facebook/Twitter, etc. There are maintenance guys, aka NOC monkeys, and there are engineers.
Eventually, everyone will need to be an engineer and that is where systems administration is going with the devops movement.
Here is a presentation I gave on it and you tell me if sysadmins are going away. http://crunchtools.com/wp-content/uploads/2010/04/DevOps.pdf
On two occasions I've had the unfortunate displeasure of working with him on web-related projects (we share similar hobbies and our local communities need web services of various kinds) his approach has been vile, Rube Goldbergesque concretions of Perl & shell scripts.
This is absolutely THE GUY I'd call if I needed some advanced logfile parsing in a hurry, but when it comes to actually developing web stuff the guy's totally in the dark. What's worse is his only metric is "It works and it makes sense to me" so he's content to make a complete hash of a development project.
During a recent conversation he admitted to having no real grasp on basic HTML and hadn't heard of CSS. The thing that kills me is he's convinced he's qualified to run websites for our local groups and absolutely refuses to accept input on the subject.
Oh well, the guy's also an ace whitewater kayaker and one of my favorite paddling buddies. Considering this guy will probably save me from drowning one day I can look past his failings as a "developer".
In the same way, the demands shifts from one sort of systems administrator -- the traditional sys admin who racks and stacks machine and manually edits config files on a UNIX system, to another: one who approaches the problem from a computer science and engineering point of view (using software such as chef/puppet, developing custom tooling, working closely with application developers, etc...).
All of this is not news to me: I've been doing this kind of work in 2006 at Yahoo before it had a buzz word ("devops") attached to it (these days I've shifted to being "pure" software developer). Broadly speaking, we had three kinds of operations personnel: those working at the physical layer, the network layer and those working between the operating system and the application layer.
Majority of the upper layer work was either writing code or working very closely with application developers. Thanks to these tools we could deploy and re-deploy machines within the matter of minutes (with no virtualization involved) and handle servers arriving by the truckloads.
I'm glad that Amazon and others have made the lower layers a publicly available service, so that the people preferring to work at the upper layers _but would prefer to be in a startup_ (rather than a corporate behemoth) could concentrate on the work they find fulfilling.
Wait, that doesn't sound right.
The points stands, though: I really don't want to do that stuff and he's welcome to it. And I seriously doubt the whole world is going to use someone else's servers... There will always be companies that don't trust 'the cloud'.
I think it was The Daily Show, when John commented on media headlines.
For me, a nice service like PubNub would run us nearly $300/day. Compare that to a simple EC2 m1.large at $240/month.
Visibility comes most of all with DBs where you want to manage lower level settings. Things like MySQL configs, disk volumes and their configuration, etc. Good luck tracking your IOPS on a hosted service, or since they're likely cloud based, getting good throughput/latency levels.
Overall though, sysadmins are changing into devops. They're the ones who connect the dots between what the app is doing, what scale it needs to run at, and the systems need to support those. As things grow, need management of all the moving pieces. Then comes monitoring minute aspects of the environment to ensure performance and stability, paging when something goes out of bounds, etc. And as you grow, minor changes or upgrades can have a huge impact or require a lot of roll out, so need to test and benchmark several aspects.
Sysadmins in the small scale may be less important, but they're becoming even more critical as you grow. The nice part is the tools they can leverage are growing so you can do more with fewer hands.
The majority of sysadmins that I know works in the financial market, telecommunications industry and ISPs, mainly in data centers, helping developers who does not know about the infrastructure or as network operators.
For me, it was the best decision, as I enjoy programming much more than sysadmin work. I get my kicks out of creating new things.
Valuable experience though.
Especially in organisations which handle data of a sensitive nature the sys-admin (or whatever his/her role will be called) will have an important role in the organisation (think especially of organisations which have to conform to HIPAA, PCI, or other data protection regulations).
The success of an organization will in these Internet times be determined by the value of the (customer) data it holds and the ability to extract business value from that data. A data admin is therefore an essential task in any organisation. The future role of Sysadmins will therefore move in that direction in my opionion.
Disclaimer: the writer is an IT Architect at a large DataCenter Services Provider.
It would take a pretty major shift in attitudes for medium and large companies to start trusting vendors with the kind of data we're talking about here.
I've only worked on ~10 paid development gigs and one of the first ones was a pharma research company who hired my team to create an interface for a truckload of sensitive data after nothing more than a phone call and a quick NDA. This pattern has continued in a few subsequent projects, often with no NDA or contract whatsoever.
A great many companies stop caring about privacy the second they can make or save a few bucks, questionable legality or ethics notwithstanding one bit. Businesses in general are fortunate that MOST software engineers seem to be honest and good people, because they regularly hand over the keys to the castle, often without even realizing it or considering the potential implications at all, and to be honest I'm actually quite glad it works this way. For whatever reason, we as a profession have apparently been deemed highly trustworthy, and I much prefer this situation with all its risks to a world of bulletproof firewalls and bureaucratic red tape.
There's a world of difference between hiring contractors to come in and develop in-house systems and betting the farm on a 3rd party vendor.
"Cloud-Hosted Service for Real-Time Messaging"
"There is no guarantee that all messages published will be received in the same order that they were sent."
... So not really real-time then. Disappointing, since I was already trying to think of ways to use it.
1. How to glue disparate systems together.
2. Determine what services to standardize on.
3. Prevent business types from choosing tech sizzle over steak.
4. Translate technology to human readible form.
So long as control and management of information is important to businesses there will be sysadmins.
I suppose this is a win for people that just want email, or documents, or calendars, but for anything else, you're stuck maintaining your own data in the cloud.
Back in the late 90's I had a choice. I could go be a programmer or I could go be a sysadmin. I really love programming. I in fact spend my "free time" (as much as a father of an infant and toddler has) programming. However at that time I noticed all the noise about how all the programming jobs were being outsourced.
I made the conclusion then that they could outsource the programmer, but they still need someone to run the infrastructure. I think that will always be the case. I've worked for business focused, rather than technology focused, companies my entire career. OK, 13 years in I've only worked for 2 companies so I can't say I have a ton of exposure, but still... My experience is that the business user wants things done a certain way and a lot of my time is spent engineering solutions to meet those requirements, which often change several times during the project phase. Cloud solutions currently are a pretty well defined box, one that I don't see a lot of business users being comfortable with the restrictions offered.
I do see some things going to cloud. Email for example... please take it. I could get more work done not having to do restores because someone decided to make a POP connection to their mailbox with their phone and deleted all their email on the server. However most of the other solutions I'd see going to the cloud are going to have to go onto basically virtual servers like what Rackspace and Amazon EC2 instances offer. And who's going to know how to manage, configure, and keep those security updates going? Oh yea, the guys who've done it on physical infrastructure for the past few decades. So basically if a company goes that way then I'll have less warranty replacements to worry about.
I think perhaps the cloud is going to change the nature of a sysadmin's job, but the profession isn't going to go away. Heck, I don't even know that I've ever been a pure sysadmin. Seems I've always had to do some programming, engineering and dabble in network the whole time I've been doing this job.
Lastly, I support my company. When there's an emergency, I am available to handle it. I'm not busy managing several other companies disasters as the same time. My business can also dictate my downtime. Oh, you're having a conference in New Zealand during the time we have our normally scheduled email maintenance, well let me just move that window for you so you can get your job done. Don't forget, we're a service based industry and you're going to get a lot better service from someone you pay to be dedicated to you rather than from a company that supports a few thousand customers. You don't have to wait in line for this car wash.
Although it might make for a more interesting sysadmin job for those that remain.
All else is not equal. The economy, and especially the tech sector, will grow.