How a sysadmin can become a programmer | Hacker News Reader

How a sysadmin can become a programmer

cuddletech.com

75 pointssea6ear79 comments

Great advice except for this bit:

"Step 3: Scripting isn’t programming."

I have to disagree that scripting isn't programming. To me it is no less programming than using a library, except for you are interacting with that library through stdin and stdout instead of a direct function call. If there are no libraries available for a single user tool, then execing is your only option. UNIX is designed to pipeline many different small tools that do one thing very well together. What's the difference between doing that at the command line vs doing it in a script? Sometimes it's just the most efficient way to program something. And with the use of exec tools can get quite powerful quite fast.

> What's the difference between doing that at the command line vs doing it in a script?

Testing. Clean consistent APIs. Dependency bloat.

Because my 50-150 line bash scripts need that sort of overhead.

They should also incorporate the currently popular design patterns, dependency injection, etc, lest someone else be able to figure out what they do just by looking at them.

Perhaps not. But that is why they are scripting and not programming. Programming is a large discipline. Writing code is small part.

Scripting is not programming in same way clicking through install wizard is not system administration.

There is still a massive difference between developers and sysadmins even in the devops era. You can make a caricature of both professions to point out the difference.

A sysadmin and a developer walk into a bar. They decide each to make a website doing order management and return in a month to demonstrate both solutions.

First the developer gets his turn. There's a massive version history, which is shown. After 15 minutes of browsing through commit messages and pointing out test cases and "look how easy it is", everyone in the cafe requests to be shown the actual site. After pointing out there's more LoC dedicated to unit testing the code than there is actual code, the programmer gives in and opens a web browser. "Could not connect to site". The programmer forgot to start his solution's web server. He logs into a shell, gives the command. After 3 tries he gets the command right, and it won't start. It is quickly noticed that there is a bug on the top of the main function preventing it from starting. He quickly fixes it. The website has every feature you could possibly want and more than a few you wouldn't want, and looks beautiful, if a bit "full". And every individual feature works beautifully and perfectly as designed, and then 2 or 3 features have to work together and an hourglass appears. After 5 minutes they give up the wait, and the programmer digs into the code and concludes that it was completely unreasonable for the sysadmin to ask for a demonstration of picking a product and while a mail notification service was enabled, something which was never in the requirements ! Getting a report out of the system takes the better part of the day. Before the demo can go further the web server crashes and not only did the programmer lose the entire order database, but the versioning system, including all the code, and surprisingly the crash also managed to take out the entire contents of the programmer's gmail inbox, X-COM savegames and his drink is suddenly empty. Only the drink's disappearance can reasonably be blamed on the sysadmin.

Second, the devops/sysadmin turn comes round. His website has been running for a long time, in fact he says he didn't bother restarting it to add the last 10 features he added to the site. The programmer (and everyone who isn't the sysadmin) has serious trouble distinguishing the website from a terminal window, or indeed, from an unpowered laptop. Tiny gray letters on a black background list the orders, as confirmed by an inspection of the sysadmins laptop screen with a magnifier glass and extreme brightness settings. They are preceeded with 10 letters that can be either small caps, or capitals illustrating the various properties an order might have. Oh and it can also be one of 2378 other letters that indicate various things, but even the sysadmin himself has forgotten most of them. There is one test, which starts up the system, and runs 1000 orders through them. The "RoundFloat" function in the code turns out to return a string, "dog", but is never called. There is no reporting functionality, insists the programmer. The sysadmin disagrees. Look ! echo 'cat orders//??d.2013..xyzz2??3?? | sed "&()&^#@\&^\\@#" | grep | perl -pe "lqwtq8(&()lj;lkd" | mail jef@boem.com' >> /etc/cron.daily adds daily reporting of all orders in the asian region over $220 to the system ! The sysadmin insists that this is simple, as he got it right the first time. When a person standing at the bar is asked to enter an order into the system, he tries for 10 minutes, then knifes the sysadmin in frustration, before retreating to an optician for damage control. The "web server", which turns out to be a bash script, has never spent more than .0003 picoseconds serving the page. The page ? Yes, there is only one after all. Why would an order tracking system need more than one page ? It turns out the webpage page is text/raw, not text/html, after which a baffled sysadmin enquires what's wrong with that. An inspection of the versioning system reveals 3 commit messages "start", "middle" and "end", and nothing else. It's replicated to 5 different datacenters, though. Look ! An offsite backup was initiated as soon as an order was entered. The programmer, still angry about his crashed server, takes an axe to the sysadmin's laptop, which surprisingly remains operational after being hacked in 10 pieces, set on fire, and fed to a local dog. The sysadmin declares that he's very proud of the system's features, you know, both of them.

The whole thing ends in a barfight between the two, the sysadmin insisting that a non-working website has 0 features, even if the programmer implemented 3000 "well-tested" ones, chuckling at the "tested" word. The programmer was heard saying that a single black screen does not an order system make, before breaking a beer bottle and attacking, with help of the person that was asked to enter an order into "the black hole".

Those three seem to be features of software development rather than programming.

To me programming is writing a program.

There are plenty of professional programmers out there that fail all three of those requirements.

A true Scotsman would never take the position that scripting is coding.

Because procedural coding is invalid.

I guess it depends on a definition of scripting. There's a huge difference between someone writing a script which has nicely defined logic and is maintainable... and writing a three-liner with 5 calls to sed, one to awk, dumping temp files all over the place and using non-verbose regexes that only the creator understands and only at the time of writing them.

There are lots of examples of both kinds of scripts. But in my experience bending bash to what you want to do and making everything a quick solution for immediate issue is a bad idea. At the same time I see it in many places where pure sysadmin skills were applied.

I'm a sysadmin who used to be a coder. I know a lot of people who have gone in either direction.

The reason I stopped coding is that I got into it to solve problems, but I got tired of spending what felt like 90% of my time writing unit tests, wrestling with frameworks, and solving problems with upgrades to the huge webs of dependencies for even basic projects. In other words, I got fed up with what coding has become. It's no longer hacking. It's toolchain management.

I'd always been a full stack developer, so I just started focusing more on the sysadmin side and gradually transitioned over there. I can manage toolchains just fine if I want to focus on that. My coding experience is handy when we're trying to troubleshoot a bug that only shows up on production ... uncommon, because our dev and QA groups are GREAT, but it still happens. I not only know where all the logs are, but I'm able to read the code and make sense of stack traces from the different languages we use.

Those who think what I've done is ridiculous don't understand how scripted everything that happens in the sysadmin world these days. Everything in infrastructure/operations is scripted except the troubleshooting -- and even that stuff gets scripted as soon as it's done so that it doesn't pop up again. Yes, the native language of these tasks is bash, but ruby and python are becoming really popular. So's haskell, for those who can speak it. What's disappointing me right now is that all of the chef stuff we're doing is starting to get complicated enough to have a big toolchain (thankfully managed right now with bundler) and starting to grow unit tests... Didn't I make a career change to get away from this crap?

I think I'll go be a DBA next. ;)

That's a weird association between "unit tests" and "toolchain management". Did you feel it was a tedious part of development or what?

To me, unit tests are the ultimate in tedium. I appreciate the value, but writing them is certainly about the dullest part of the job.

Thanks for the response :-)

I know a fair number of developers who find ops pretty crazy, considering that a good chunk of it can be automated. So I suppose the other side of tedium is the unnecessary risk in having to hand-perform things repeatedly.

there's more to ops than just deploying. it's about running a smooth environment from end to end. automation gets you 90% there and give you the time to focus on that much more involved 10%.

Fair enough.

What do you think about Anderseen's "Software Eating the World?" Wouldn't it make sense that at some point in the future, "running a smooth environment" gets eaten by software too?

houk got most of the answer. The other answer is that ops, unlike dev, still gets to do things the fast way if it makes sense. So if it's faster to hit up and enter a bunch of times? Do it. Dev doesn't get to make that decision most of the time these days, and that's just sad and completely against the spirit of hacking.

There's hacking, and then there's polish. You can't make a product out of a hack. Though some companies offer Hack Days to get things going.

> tired of spending what felt like 90% of my time writing unit tests, wrestling with frameworks, and solving problems with upgrades to the huge webs of dependencies for even basic projects. In other words, I got fed up with what coding has become. It's no longer hacking. It's toolchain management.

I'd add to that, dealing with programmers who insist all of this is absolutely necessary, as if software didn't exist before these advancements. Meanwhile, it takes them a couple days to program themselves out of a wet paper bag.

This whole article is full of prejudice, narrow-mindedness, and over-generalisation. I'm not a sys-admin myself, but I have known some, and they certainly didn't fit this stereotype.

Yeah, I worked for 2 years as a sysadmin, and now I'm working on building a website in one of those scripting languages that apparently don't count as programming, so perhaps I have a better perspective than most on this. The strange conceit of this article is that all sysadmins apparently want to be programmers but just lack the self confidence to actually do it. I think that this is pretty wide of the mark. If anything, sysadmins know how much more stable and reliable mature tools are than the sort of half baked solutions the author is recommending that they push themselves to develop. When you spend your entire day supporting fragile solutions, your first inclination is not usually going to be to put more of those into the world.

now I'm working on building a website in one of those scripting languages that apparently don't count as programming

The article didn't claim that scripting languages don't count as programming. Despite the heading "scripting isn't programming", it doesn't even really claim that scripting isn't programming: it claims that if you switch from shell scripting language to a compiled language, but you still do most of the stuff by spawning child processes, you're most likely using the wrong tool for the job.

Forgive me for countering hyperbole with hyperbole. She did baldly say in the main content of her article "Scripting isn't programming". It's not a hyperbolic title, it's in the body of the article. She also doesn't counter that assertion. The article is hyperbolic through and through, and I don't care to argue its finer semantic points since the original article was neither nuanced nor subtle.

The strange conceit of this article is that all sysadmins apparently want to be programmers

Speaking of conceit, the article didn't actually say or even imply this. You're projecting. It mentioned a line 'most' sysadmins use, then talked about some strategies you can pursue if you want to 'become a programmer'.

Let's just, for a moment, entertain this radical notion that many sysadmins are actually programmers that just don't want to do software development as a profession. I know that, for most people here, this is very strange: who would choose to avoid the exciting world of programming having the skills to do so? But indeed, it happens.

For someone that likes to solve problems, 95% of software development is mind-boggingly boring. It's mostly looking at the same codebase day-in and day-out, while having to deal with shifting requirements. All of this to end up with some application that you mostly wouldn't use yourself. This, if you're lucky that your project doesn't get cancelled before it sees the light of day, or it doesn't get delayed so much that you get so bored that you move on before seeing it to completion.

And the consultancy... Parachuting into projects at clients and being extracted by helicopter when "it's done" (which is usually when it meets minimum requirements to receive payment, not when it is at all in a state to be proud of).

Really, 95% of software development isn't glamorous stuff at NASA or the next big social network(1).

Systems administration, provided you are self-motivated, can actually code your way out of a hole, and are willing to deal with the occasional user, provides plenty of space to solve problems with programming. With programming, not IDE and framework juggling.

Sure, it can be boring at times and can be stressful at others, and you have 95% chance of being in the exact same environment where you wouldn't like to be coding in. But at least you have more autonomy and can actually solve problems.

I'm a programmer by training and a systems administrator by profession. I would most likely rather be programming than taking care of systems and databases, but only if I found something in those remaining 5% that I would actually enjoy doing.

Otherwise, it would just burn me out from programming altogether, and I like programming...

(1) Actually, developing the next big social network is probably as boring as everything else I said. Who in hell wants another social network?

Well said. Programming is solving problems with (hopefully) thinking and problem solving skills that improve every time you do something. Differentiating between coding/engineering/programming/scripting is all a moot point where it counts -- creating value and solving problems.

No matter how perfectly you do or don't architect something, it will have a failing point, just like the limits of hardware when you buy them. There are some very general thinks you can do to be kind to your future self architecturally, but just as many you can do to hurt yourself.

I think edw519 said it best, clever architecture can be far more important than clever code.

Maybe it's better to know avg programming and avg administration and use both skills to do something really interesting? Advantages: use existing codebase, fill holes with own code and set up infrastructure for a project. With a smile :-) Disadvantages: mastering specific areas, but this could be personal preference because having more then one person collaborating on the same thing can be real painful. It all depends.

Hi, I'm a systems administrator. I'm reasonably senior. I can tell you that pretty much all sysadmins code. In overlapping sets, some are good at it, some don't need to be good at it, and some like solving problems that way. But being a sysadmin is a different job from being a software developer, and people who are good at one and not the other are much more common than those who are good at both.

The nature of systems administration work falls into two broad categories. The first is small problem work: a user has a problem, a machine signals an alert, something isn't working. The second is, of course, large project work: users need a new service, the alerting system needs to be replaced, new infrastructure needs to be built. Most of the time, but not always, the small problems need and have immediate solutions, and the large projects are not as time-critical and require some amount of research, budgeting, selection, configuration and support before they can be deemed complete.

A sysadmin who has specialized into a particular problem domain will need the skills appropriate to that domain. Sometimes that means picking up a new configuration syntax -- which may be a complete language. Sometimes it means building better communications skills, so that you can diagnose what your users are complaining about. Whatever it is, you can assume that the people who specialize in an area that doesn't require much programming, don't do much.

Generalist sysadmins, on the other hand, need to be competent at a wide variety of skills, including two or three languages. That doesn't mean that they need to be UX designers (but they need to recognize a UI failure) and it doesn't mean that they need to be database administrators (but they need to understand how to back up and restore the databases they support, and figure out the replication system and find single points of failure). They do need to be able to write short programs to automate what they would otherwise do manually, to figure out what the debug logs are really complaining about, and to understand computer systems from electrical and environmental needs through virtualization systems and configuration management.

Finally, there are sysadmins who are also developers, who announce that they don't like any of the existing mail programs and so they will write a new one; who determine that there's no configuration management system that tracks and removes subsystems as cleanly as they would like, and so a new one will be needed; who write filesystems or music organizers or browser extensions because that's what they need and want and are good at.

The industry, nebulous quagmire that it is, needs specialists, generalists, and dev-ops hybrids. Making false generalizations merely reveals a bizarre lack of awareness.

I'd just like to mention something about specialists vs generalists. Over my career, as a sysadmin, I've noticed this trend. At smallish organizations (i.e. startups), you'll see almost exclusively generalists, because the company simply is not large enough to have a team of specialists running around. As the company grows (think fortune 500 companies, universities, government, etc), these generalists will have to make the transition into specialists, say for example into, security, networks, email, linux admins, windows admins, storage, racking/stacking/cabling, etc. You might even have these enterprise architects popping up.

I have also seen less sysadmin programming (whatever that is) at the larger companies, because they will purchase prepackaged products that fit their need.

In many cases the "prepackaged product" purchasing decisions are being made by managers who won't actually use them. That's why enterprise sysadmin jobs suck so much.

To counter, I worked at a small web hosting company between 2007 and 2010 who were responsbile for creating Europe's first cloud computing infrastructure and the worlds second, according to them at least.

They had about 9 sysadmins go through their doors, a maximum of 7 at any one time with a max of 4 coders. Of those system guys I'd hazard a guess that around 50% of them couldn't code at all and only really two could get anything beyond procedural scripting, if that.

Not to disrespect these guys because some of them were between good and really good at what they did do, but there were also decisions such as to rely on expensive SANs and proprietary software to tackle issues that could have been and later were replaced by internal code because, in my opinion, these guys were unable to comprehend automation on a scale that a developer could unless they already knew of software available to do such a thing.

After that I worked for a massive hosting company with much more segregation of departments and as far as I'm aware operations didn't code despite the deparment being a lot larger than the previous company.

It's funny because back then I was arguing with both these companies that operations should be driven from an automation stand point utilising devs skills but it wasn't exactly a great point since I was the only developer on staff for much of that time in the first company and the second wasn't interested at all.

For the record, I'd like to be a devops but I'm pretty sure I get shut down from those roles from a percieved lack of operations experience and buzzwords. Much of that because I'm actually honest and my experience of sysadmins are that they were gatekeepers unwilling to so much as let me login to some of their systems. Meh.

I'm sure theres a lot of sysadmins out there who a very capable software developers but I think it's a stretch attempting to call often simple configuration languages and extremely short glue scripts programming. But hey, as long as you're adding value, who cares what we're calling it.

Can't add much to the above, just wanted to lend my support to what you've said:

I had to learn and maintain the network, equipment, software, and the code to all work together in one solution. They may call this full-stack developers now, but in my circle of friends I grew up around in technology, it was the only developers that could learn and work with the full stack that got the most done.

SysAdmins from the past years might not be able to code, but I can tell you this much: 7+ years ago, there was very little luxury of being a coder or sysadmin only if you were close to solving problems.

Most sysadmins aren't LAMP (or LNMP, or LNPR, or whatever the flavor of the day) programmers, who know how to make a todo list "pop". They aren't javascript ninjas, who actually know how to interact with the DOM. They don't know how build models of black holes, or self-driving cars, or make enemy sprites hunt you down in an interesting way, or debug a GUI.

Sysadmins do program. Their programs tend to be flatter (scripts, not frameworks), but that doesn't mean they aren't programmers. They probably can't build great frameworks, but not a lot of programmers can (but sadly, they try - every big corporate project seems to grow into a badly thought out framework).

If you pick any domain that's not "compiler design", you can probably make a good case that they aren't really programmers.

Most software engineers couldn't do any of those tasks to the typical, silo-enforcing business asshole's liking (i.e. it needs to be done yesterday). That shouldn't matter. What matters is the ability to learn.

Sure, most people working as sysadmins can't code. Then again, most people working as programmers can't code either. People who can really write good code are few and far between.

The very best sysadmins are also good coders, and the very best coders are also good sysadmins. The foremost experts on any operating system are the developers that work on them after all. You can never fully understand an operating system without reading the code, and you can't write the most efficient code if you don't fully understand the entire stack the application runs on, operating system included.

As for using tools instead of writing code, what it comes down to is what's the best solution for the problem at hand. A good sysadmin doesn't need to memorize ways to use tools; they know each of the tools well enough to make full use of them. If a task can be done with a one-liner, and has a fixed amount of work to be done, why would you spend more time to implement the same in a programming language, when the execution time is insignificant relative to the development time? Considering the slower higher level languages that are popular nowadays vs the standard tools written in low level C, the execution time may actually end up slower despite the execution savings in running a single process instead of many.

> The very best sysadmins are also good coders, and the very best coders are also good sysadmins.

I can agree on the first, but the second while surely true in cases is maybe wrong for people doing something completely different, like are more into the academic areas, in really abstract levels, doing big data stuff in completely managed environment.

Think about planes. Someone who crafts and maintains them and is close to the metal, maybe is a really good pilot too, because he knows planes in and out, but while really good pilots probably know a lot about planes, they may be focusing on stuff that is really, really far away from crafting one. Still they could be among the most excellent pilots.

But then who is who, might depend on whom you ask.

I just really find it funny and that's something that kinda came across in the article, how because sysadmins are frequently good in a field of programming that by programmers is considered a field of masters, which are operating systems and all this low level stuff that many programmers have to get to at some point. In a way sysadmins have to do and shell scripts are pretty high level. But it also really shows how it is a lot of two different mindsets.

I've never met an openbsd/freebsd, solaris or old school unix sysadmin that didn't have a total mastery of C/++ and scripting like perl. By total mastery I mean debugging and even reverse engineering binaries looking for something that was eating memory, and often completely rewriting some major program parts to add security like old smtp and sendmail. The Sr admin I worked for could quote K&R by memory and was a much better dev than the software engineers there just he was a total neckbeard and didn't want to deal with "politics" so confined himself to the sysadmin dungeon. He made more money than them too

Just look at openbsd-dev mailing list. Most of those guys are sysadmins, like the guy who wrote OpenBGP from scratch, the pf maintainers... basically all Berkley forks were created by sysadmins.

Do you see many OpenBSD or "old school" sysadmins? I mostly see kids that haven't touched anything but Debian and Ubuntu, think RHEL is too old school and never heard of fork(2).'

Good sysadmins obviously exist, but in my experience they are a shrinking minority.

I think sysadmins period are shrinking. Most companies are just signing up to cloud products and using feudal security. The only jobs I've seen locally here advertising for sysadmins required that you had fluency in C/python. Everything else is just a $14/hr desktop tech support position.

Exactly. Mastering a language is function of time, but trying new language is not a bad thing. I'd even say that sticking with one language/paradigm is bad thing, because it's possible to master more then one language and you could get new perspective by learning new stuff (and finally maybe finding something useful from all those hipsters stuff).

As a set of bullets, I can agree with parts of this list, but the supporting paragraphs of weird assumption and baffling generalization, not nearly as much.

The phrase "I'm not a coder" can mean all kinds things coming from a sysadmin, context is import.

If a sysadmin truly lacks the ability/confidence to write useful code, I'd expect has more to do with the typical nature of the job and organizational divisions than prejudices, hang-ups or fickle behavior of the sysadmins themselves.

A role which is too often reactive doesn't leave much time for development. Likewise, development towards the goals of a sysadmin (automation, insight, reliability) too often lead sysadmins to unemployment due to misunderstanding as the relationships between sysadmins who "don't do anything" and systems that don't break.

I expect new businesses, particularly the type common on HN could be exceptions, but in many organizations the preferred language of system administration and those of software development don't align. As a result, serious sysadmin code written in say Python as opposed to corporate standard of .NET/Java is putting the sysadmin on an island and likely to be seen as a threat/liability.

Couldn't help but write a response - http://joerussbowman.tumblr.com/post/54602688494/why-sysadmi...

This divide is a really fascinating to me. I feel like I'm watching a National Geographic special.

"We don’t want to learn the latest IDE and develop best practices for working with a version control system with other developers. Code reviews, strategy meetings, QA reviews… what? No thanks, let the programmers get that stuff done. They have 7 hours to commit to this project today, I have 45 minutes and that’s only if all my KLO goes well."

Why do you feel this way? (Not a rhetorical question). Looking at it superficially, I would guess that:

(1) You don't like the engineering part of software development, finding it too tedious or something? (2) You feel like you're more badass for having to get shit done in a much smaller time frame.

I read through some of the other responses and it seems to fall into this general pattern.

Not the parent and I don't totally agree with his position, but I can certainly relate.

>(1) You don't like the engineering part of software development, finding it too tedious or something?

I find the engineering of software fascinating. I've found the glacial pace and ceremony of most software development teams I've been exposed to unbearable.

I can't imagine this is intrinsic to software development, more likely a clash of cultures and eras.

For example, during my longest stint in a sysadmin role (~10 years) I supported and migrated between a continuously shifting mix of OS/2, Windows, AIX, Solaris, NetWare and all sorts of Linux speaking SNA, IPX, TCP/IP on EoC, TR etc.

I did this while accommodating and often clashing with internal developers who had been working the same codebase the entire time.

Again, I don't mean to present this as something necessarily true of all development vs administration as a whole, but I've seen it and heard similar experiences related enough to think that it's not uncommon.

>(2) You feel like you're more badass for having to get shit done in a much smaller time frame.

Well, "badass" seems a bit condescending.

When a sysadmin needs to get things done in 20 or 45 minutes, there's a good chance his job depends on it. Though they generally wouldn't choose to put in positions like that, I have found that people who excel in the role are the ones who can stay cool, thrive under intense pressure.

Interesting. Seems to me, the main thing is the pacing.

So what about, for example, video game development? It's infamous for development to be in continuous state of crunch time.

Coincidentally, one of if not the best programmer I've worked with was previously a programming sysadmin for a big name in the games industry.

His stories corroborate what is often heard about gaming companies - that the continuous crunch time isn't necessary so much a matter of exploiting an endless supply of young, enthusiastic talent.

I'd certainly see myself having issues with that kind of environment.

The fast iterating development culture of many Internet-era companies certainly sound nice, but never having worked in one, I'm not sure how of the hype to believe.

I don't work for a game company, but yeah, I agree that the crunch time doesn't work well. I've read articles about the correlation between that kind of crunch time programming, and the general maturity of the artform. That is, the guys (I remember the industry being dominated by males) who raise a family and enter that stage of their life end up dropping out of the gaming industry. Thus, games tend to be made by guys who haven't had that life experience, willing to go into crunch mode, etc.

I brought it up though as an example of pacing. I'm trying to discern whether it is the pacing itself, or whether it is the programming jobs you have tried have been with organizations that have a lot of red tape, or whether it is simply the engineering discipline that you prefer not to participate in.

As for the Internet-era companies, the fast iteration is something you tend to see more in startups rather than startups that have matured into big business. Iteration for the sake of iteration itself doesn't work very well either (John Boyd "fighter pilot agility" vs. Agile Manifesto). It's also where devops is coming out from, and proliferating back into mainstream IT.

No. I personally do enjoy engineering tasks. As an ops guy though I find I look at things a bit differently and often end up as a resource for the programmers to help them understand how to fit their apps into our existing infrastructure. I personally, as a hobby do application development, I wrote and manage unscatter.com. It's not a case of I am superior to our programmers, it's a case of the expectations of me are different. It's not my job to make great programs, it's my job to make sure all the great programs run correctly. I'm expected to keep Exchange working, apache working, manage dns and a few other systems to to date today and tomorrow a whole other set of services to be managed. I rely on our programmers to have the time to focus on one thing so I can spread my time managing everything else.

Fair enough.

The way I see it, in:

Age 0, all computer users where actually users and sys admins and programmers and EEs, then, when computers stopped being room-sized monsters, in

Age 1, all users were users and sys admins and programmers (the Lisp machines era is probably at the end of this age), then, when software became actually usable by most non-programmers, in

Age 2, all users were users and sys admins (the UNIX era falls in this age), then, in

Age 3, users started to be only users - or, more exactly, people who were not programmers and sys admins started to actually use computers on a large scale (the current era of Windows and Macs and everything else) - since sys admins separated themselves from the users group long after the programmers have, they are "philogenetically" closer to users than to programmers ...also, this is why I think the "Linux desktop" dream took so long to rise up, becase Unix systems by design ask users to at least be sys admins too, and reward them to the max when they are sys admins and coders too.

Now, if anyone has any idea of what comes next, please share the enlightening thoughts :)

The only witty continuations I can think of is: Age N - humans are no longer "using" the machines and they haven't done this for a long long time (well, they do, but only in the sense that a plant uses the sun, water and earth to survive), they are being used by them, grown in "human natural reservations" for scientific study purposes and historical education purposes for machines to remember how the beings that preceded them were like ...but hopefully someone else has a brighter view on the future.

The way I see it is: Age 4: All users are sysadmins or programmers, because the ones who are not have moved to a tablet.

I don't see tablets as anything more than just another PC form factor... a secretary that needs to type fast will use a tablet with a keyboard and a programmer just as well. Once we go past Apple's "tablet as an appliance" dark age, as Microsoft gets more market share (yeah, I'm siding with Microsoft here (and against Google and Apple) and I think they are, willingly or not, pushing things in a better direction by having the same platform for both tablets and PCs, even if their product quality seems inferior) and Android get more and more features and starts to look like a "real OS", programmers and sys admins will be able to use tablets just as well, attaching keyboards to them when needed and plugging them into a large screen at their desks... heck, I can even imagine programmers typing code in the air on the grand-grand-...-grand-children of Google glass, with a pair of mechanic-feedback gloves or something.

Interesting, when I wrote my first comment I was thinking that PCs would stay the machine where programmers would write software for devices dedicated to the large public (tablets, smartphones, smart glasses, smart watches, etc.), but after reading your answer I realize that I may be wrong, and as you say, those devices could very well be used as development machine.

I don't know what's the state of the coders tools for smaller devices, but that's an idea PG would like to fund: http://ycombinator.com/rfs5.html

I don't think making dev tools tailored for a specific form factor or even platform is a good idea. You can use something like Ms Surface Pro to do dev work right now with the tools already available even if they're far from perfect - this is why I like Ms's "vision" on the whole tablet saga better.

What I'd really love to have (and even I'd even pay good bucks for it!) would be a browser-based-polyglot-IDE, maybe an IDE implemented as a browser extension, that would simplify the fucked up workflow of web frontend dev (none of the existing editors, IDEs or anything else properly handles the ungodly tangle of HTML/CSS/SASS/LESS/Javascript/Typescript/Dart/CoffeeScript...) and be totally platform independent so I can use it on any device and just tweak some configs to adapt it for a particular small screen size.

I think "RFS 5: Development on Handhelds " has a wrong perspective, you don't want to think about development on handhelds, you need to have dev tools that are so platform independent and form factor independent that they work on anything (code-mirror seemed a nice base last time I checked, but compared to what's needed is like comparing the linux kernel with full fledged desktop linux distro...). And if you think about tools for developing native apps, I think this is "sailink a sinking ship" anyway and on top of that you could always be put out of business by Apple or Ms making an Xcode or VS version optimized for handhelds and giving away for free and "express" version that will be good enough for most.

Anyway, this has veered seriously off topic, but dev tools are something I care about and I couldn't refrain from commenting on that :)

You missed the mainframe era where from at least the 1970s forward you had office workers by the score who were only users.

You're right, I think one may say that mainframes did the Age 2 -> 3 transition first, as it was kind of necessary for their use care, and then PCs recapitulated history...

small point: the author isn't saying "scripting languages aren't programming languages", nor are they saying "you cannot write programs in scripting languages" - they're saying that scripting - a particular style of program-writing - isn't programming, or put better "proper programming" which is arguably true, especially if you're writing bash or IPython scripts, where you have access to and make heavy use of shell commands.

That style of programming is very very different to regular compiled programs, and practicing one and never the other will lead to a bias, and relative incompetence in the other. This is why I'm rubbish at chaining unix commands together, because whenever I'm confronted by a problem that might need something that complex, I just use python. I have the reverse problem to the rhetorical sysadmin in the article.

The point of sysadmin scripting is to automate stupid and repetitive jobs. They are payed for that kind of programming. Also, it can be viewed as kind of hacking -> word that is not popular in most of enterprise environments. You know it works, leave it alone :-)

Exactly, because sysadmins have time and luxury combined with training in programming, they know how to make their jobs easiy.

Why Programmers Can't Use Apostrophes Correctly

Or control comment-spam

Some of best programmers i have met are, watch this out, sysadmins. And no, they will not do your job :-)

On the other hand most of developers' knowledge is like swiss cheese, because they don't know enough about underlying machine, they only want to deal with abstractions and algorithms, so for example trivial things like where is located config can be really tricky for them. As you can see, one could do same article about developers, but what's the point on spitting on people you work with? Ah, yeah, frustration :-)

This actually describes how I have approached system administration over the past few years. I'm just now trying to change that. I'm embarrassed to say, but I'm writing HTML for the first time -- in 2013, not back in 2003 when all the cool kids were.

OK so I'm not a cool kid anymore. But I can be again, I just have to slog through some tomes and actually build something cool. Isn't that what being a nerd is all about?

The cool kids were writing HTML in 1993.

http://www.w3.org/MarkUp/draft-ietf-iiir-html-01.txt

Hi, I have been a Solaris/SGI admin (2 years) and software engineer (10 years) in my life. The reason why sysadmins can't write software is this: Every day sysadmin stuff like pulling cable, installing OS's, fixing printers does not require someone whose super intelligent. But real IT work-- setting up domain controllers, running a global corporate network, running a cloud service, etc takes a fair amount of brains yet a lot of companies compensate their IT departments as if they're well paid cable monkeys.

The skill sets for a good programmer and a good sysadmin overlap somewhat. But you can make MUCH MORE MONEY writing software. In the early 2000's when I was a sysadmin jobs in my area were paying 40k-60k a year. Programming jobs were paying 60k-100k. So basically, poorly payed IT positions select out anyone who could actually do a good job at them because those very people can make more money writing software

>SysAdmin’s are purists and have a lot of ego. Using a module or library is akin to cheating.

I'm a programmer and I know how this feels. It's not about "cheating". To me, I view it as "I can't be bothered to learn /how/ this works, I'll just use a library". I can make a half-assed HTTP library, but in production, I'd obviously use a real library. The important thing is I understand on a more intimate level how HTTP works. That is invaluable in debugging as when I ask myself "why isn't this working", I can think back to my own code and it often helps to shape my understanding. I can think how the machine thinks as I taught it how.

I'll often study other systems while building my own. It's all part of the learning process to me.

It also provides deep appreciation for the effort that's poured into libraries.

I've also done the same thing on the web; I used XMLHttpRequest and DOM apis for years before I started using jQuery. I've never had a single complaint but I've seen programmers who have gone right into jQuery and surrounded themselves with abstraction and have had no idea how the system is behaving and they get confused and frustrated. This leads people like me to view libraries as bad; they are preventing people from learning how their systems work.

Programmers today seem so lazy. I'm not really sure if that's the right word. I see people using HTML 5 Boilerplate and they never seem to ask "Why is the meta charset before the title tag?". I did. Now I know about UTF-7 XSS vulnerabilities. I spent most of my day reading up on charset vulnerabilities. With that knowledge, I patched several projects of mine. Sure, people using Boilerplate would be protected but they wouldn't really know what from.

Perhaps I just think differently to other programmers. Maybe they just want to get on with their work while I obsess over "how does this work?!"

It's not just that it's cheating, it's that it's less portable as well. A self-contained script can be thrown anywhere. One that needs a library needs to have the library taken with it. One thing that 'sysadmins' encounter than 'programmers' don't is a wide variety of systems with a varied install base. The more portable your coding, the easier time you have as you progress.

Throw in that programmers work on more monolithic projects, getting the libraries for those is (kind of) a one-time affair, but with sysadmins working on smaller, multitudinous projects, you can end up with a raft of different libraries to manage.

The real answer, as always, is 'be appropriate'. Often libraries are called for, but sometimes they're to be avoided. It depends on the use case, or even the library itself (we're currently using an abandoned one in production... and there's some crossing-fingers going on as we don't have the resources to fix it if there's breakage...)

And the flip side, why coders can't do the simplest of sysadmin tasks.

I've meet coders who couldn't spell "DHCP" let alone tell you what it does or how it works.

They're two completely different job descriptions - why should one need to do the other?

They are different but also related.

Programmers that don't know basics about network are very often poor programmers as well. Most programmers must be able to set up build servers, test environments etc that they need. Of cause the sysadmin can do it for them, but in that case they have to wait until he have time, and even then they will get poorer understanding of the system. Also a lot of programmers need to know network basics because their application has a network component. It is also important for a programmer to understand what he need to make a system administration friendly, like easy to install and upgrade, sensible logs when things go wrong etc. They don't need to be able to take over the sysadmin's job the next day, but should understand what the job is about.

Sysadmins on the other hand need to at least be able to script common tasks and write plugins for monitoring software and be able do some customization of the systems. They don't need to be able to, or use the methods for developing large complex system, but I expect a good sysadmin to know programming and scripting in some form.

Unless the coder is doing something in the network space, it shouldn't really matter. But there is a special place in hell for those professional coders who can't take a virgin machine and install their own environment on it.

Why not link to the blog post directly? This link will break as soon as the author writes another blog post about devops.

http://cuddletech.com/blog/?p=817

When a sigh will not suffice, abstract it.

Part of it is just experience. As a sysadmin I was jumping from one task to the next learning as I went. I never got the chance to be really crazily good at any one thing. No sooner would I learn something and apply it and I was off to put out another fire in a totally different area.

I can so much agree. I pretty much went that way and a lot of it is true and I wished I had done that earlier. Also I would recommend everyone to go this way, because it makes some problems easier to solve for you, even if you stay a sysadmin.

Other than that, I really wasn't a good sysadmin, actually I used to be a really bad sysadmin, but still came from that direction. This however brings me to another thing: Sysadmins aren't ever valued enough.

Everyone treats Sysadmins as second class IT people that can't code too well. However, that's from my point of view not true at all. They usually code "script" way faster. Their set of tools simply are different and designed for people trying to reach different goals.

But there is more to this. As a programmer, you have your cozy, nice environment, your nice specifications, your nice libraries, your basic knowledge of data structures, etc and you have an easy way to be like "uh you can do it that way. It's even cooler".

As a sysadmin you usually don't have anything to rely at all. You don't learn too much at universities, that can be of help and while as a software developer you basically have to just write code and make that work as a sysadmin you are dealing with unpredictable systems, have maybe a single system with multiple users, need to make sure none of these processes do something bad, yet can do everything they may need to do, need to trace down an error that is somewhere, don't have a debugger, can't just quickly run the code on your test system, have a way harder time to trace things down, but then it doesn't have to be the one system, but could be a whole network of such systems, the whole internet can be the source and you are pretty much always in the battle field, in live systems and lots of attackers could potentially attack every single thing and you basically need to know every protocol, and every piece of software, every network connection and by heart. It could be something very high level causing the problem, a faulty program, the hardware, physics, whatever and you have to try to master chaos every single day, while programmers usually just work with something simple and often see it as a super hard problem when they are dealing with some input from the outside.

I don't know, but it feels weird, when the bigger amount of math basically causes your pay to be higher, when that very thing is actually making your job easier.

I am a programmer myself. I love coding. I love my job and I kinda like to brag a bit with it, but actually compared to many other jobs, like sysadmins we are really overpayed, while way too often complain about bad sysadmins. Being a sysadmin isn't just "apt-get install apache", but way more and way intense.

Another thing that's also funny and strange is that scripting thing. The same thing that often causes sysadmin to start out with ugly code is the thing that makes sysadmin advance extremely quickly when it comes to code quality. Once they really are programmers they know that a lot of code can be complex and since they have skills in using many tools and not just use them, but use them correctly (unlike most programmers tend to) they will usually turn out to write high quality (less bug prone) code with the right amount of defensiveness and in a way that brings some kind of order into chaos.

It's a hard way, but sysadmins are maybe a bit too much perfectionists to realize that they would be good programmers and that hardly is recognized by people starting out as programmers, who would make really, really awful sysadmins btw. and no just because you set up a unix system and a firewall it doesn't make you a sysadmin.

I actually even think that the devop thing is going partly into a wrong direction. It works, but only if you are using Ubuntu. No, really. That's also not a good thing in general, because it could actually push IT backwards. Diversity is sadly not valued enough. Using the right tool can really push you forward. And while we often seek for tools used by most people it ultimately leads to stalling. Going into a direction where we have Ubuntu instead of many Linux distributions, many Unices, many Operating Systems isn't good. There are reasons for the creation of more than one and they didn't use to be as naive as they frequently are today.

Also Sysamdins are very humble. One can actually see that by version numbers. It's maybe a bit silly of an example, but they (and programmers working with them) are extremely conservative about making a version sound like it could be stable. If they try their hard to find anything to change or make better and really can't think of anything then they will maybe call it 0.9 Alpha. On the other hand, if they find something that is called 12.13.4.0 then the zero at the end makes them really suspicious about the software being too unstable (there is enough chaos already). If you know such software then it is probably written by a person somehow related to sysadmin stuff.

However, there is one thing that would make Sysadmins really great programmers: They are super pragmatic. They a´have a good sense of realism. They know what's necessary and what isn't, as long as they are not afraid of being punished for non-perfect/beautiful code. That means that they are good at writing release-ready (feature-/stability wise) code.

However none of that is true for anyone who just switched. It's just hard to learn what the right amount of quality really is and that it also is about "beautiful code".

Well, so much about my experience. If you know a sysadmin that likes to code, maybe help him and be nice. They are always complained about when something doesn't work (even when it isn't their fault) and whenever something works great it is the developers, marking people and so on who are praised.

my response: http://eleventymedia.com/

I really can't stand all of these silos that seem to exist for the benefit of MBA-culture idiots who don't understand the first thing about computers, at the expense of programmers.

"You can't X. You're a Y!" Well, fuck you.

All of this warring-camps idiocy, "hire a real X" nonsense, and all of these damn silos, just exist so a bunch of people who aren't smart enough to manage technology can feel like they understand shit. They don't. They hire some $785/hour consultant whose diagnosis is, "you need better system administration". So they go out and hire a sys-admin or two with a "track record" and the problem is "magically" solved.

Programming was supposed to be the magic skill that allowed one to move fluidly about the industrial/technological economy. It showed that you could grapple with abstraction, learn quickly on your own, and get hard things done and do them well. It conferred mobility and job security.

Then those useless "be a team player and syngerize the verticals while the rubber hits the road" monkey-morons came in and had to fucking siloize everything, just to play a divide-and-conquer game, create the sense that we were incapable of handling "the big picture", and reduce our bargaining power. This distracts us from the fact that those people aren't at all necessary.

Good business people go out and solve important business problems; the bad ones meddle in our affairs for intimidation's sake.

>Then those useless "be a team player and syngerize the verticals while the rubber hits the road" monkey-morons came in and had to fucking siloize everything, just to play a divide-and-conquer game, create the sense that we were incapable of handling "the big picture", and reduce our bargaining power. This distracts us from the fact that those people aren't at all necessary.

So true.

I worked in a largely silo-free environment up until about 2007. New management transformed an effective ~25 person team into half a dozen kingdoms of 30+ each in a little over one year.

I left and discovered just how unique my environment had been prior to the regime change. It's hard for me to imagine how any organization which has gone down this path can hope to reverse it.

It's desirable to both the managers who use it make themselves relevant and an unfortunate majority of workers who take comfort in their pigeon-holes.

Thank you.

Reply on news.ycombinator.com