"DevOps is ruining my craft" | Hacker News Reader

"DevOps is ruining my craft"

tatiyants.com

50 pointsatatiyan36 comments

Obviously the satirized sysadmin has never read TPOSANA. Automation and consistency are highly regarded in the sysadmin field. I've seen significantly more casual attitudes towards operational needs from the development side of the house.

DevOps was a term created because other titles have been watered down. Any real sysadmin has been automating deployments, updating, backups, and failovers for over a decade now. Don't claim we aren't sysadmins anymore now that it's getting interesting.

Satire. Come on people, I know it's early but this is hard to miss:

    Sure, you might get your precious “predictability”, 
    but at what cost? I mean, can you even still remember
    that rush of intrigue and anticipation you get when
    your application refuses to work on two of the twelve
    servers it was deployed to? The thrill of the hunt as
    you figure out exactly which configuration settings are
    different and, of those, which one is causing the
    problem? The sweet taste of relief that you get after
    hours upon hours of debugging finally narrowed it down
    to a rogue registry setting?

I know, right? When he suggested that DevOps people would somehow get complex server management right just because they have a python interpreter I nearly fell out of my chair.

I think this was more telling:

> “I come from a long line of sysadmins. My father was a sysadmin, as was his father before him. I have apprenticed at the feet of some of the greatest sysadmins American BankCorp has ever seen.

The lineage would mean that they've been in the business for 30+ years now?

> Satire ...

The heading on the page "geek humor and other drivel" was also something of a hint.

I've had to suffer doing web dev work in a group in which sys admins didn't allow the devs to login on production servers. This policy was for the sake of "stability" and "uptime" when in fact the sys admins lack of understanding about the code and the stack they would in fact break shit all of the time. Least productive place I've ever worked for. Laughably the biz managers kept asking why we couldn't we be more "agile" and deploy new features everyday like Facebook, why can't we do that? It wasn't even worth answering.

why would you need to have access to prod ? Your development/staging environment (not local box) should be same as prod in most aspects except capacity so your push to prod is 2 step process developer -> staging -> prod

debugging is one (perhaps the only/primary) need. if you have stuff only breaking on production systems, it may be because of a number of factors that aren't easily replicatable in a testing environment. Sometimes the only feasible way (assuming you want fast answers because of live production downtime) is to get on to production servers and look at things there. In most cases read-only access should be fine.

Ideally, this is true.

But, I've worked in places that had bad resource planning, and didn't allocate the same kinds of resources in staging. When we had performance problems, there was no place else to go.

It can be hard to convince people to take their expensive hardware and IT time to set that production system up, and now, "duplicate it so we can debug". Eventually, I think you just give up and go somewhere else.

You can't reproduce a dev/staging environment with the same traffic pattern, user behavior, and request load as to a production environment. Google bots won't crawl your dev/staging environment and you won't know its impact until you see it. DDOS attack won't happen in your dev/staging environment. Click fault won't happen on dev/staging.

Developers probably shouldn't have root privilege on production but they need to have read access at the very least. They also should have write and deployment privilege on a few production machines for testing and experiment. Whatever mess they made there should be counted under acceptable loss.

No disrespect, but after a statement like this, "...deployment privilege on a few production machines for testing and experiment.", I wouldn't let you near my servers. That's not what a production server is for.

If your admins don't understand the code and crash things - that's an organizational problem that can be corrected.

Giving developers admin access isn't the answer.

I've seen both sides - where admins didn't undersatnd the environment well enough or the application in it and broke it, and where developers didn't understand the system well enough and pushed out code releases that broke things. Neither situation should exist if things are set up correctly.

Things should be automated and documented. You need systems knowledge as part of your development process.... it's not enough just to throw a PHP guy at something..... everything needs to be architected in a way everyone understands.

Several very important standards require that developers do not have access to production (PCI comes to mind if I'm not mistaken).

You think PCI law can tell the difference between a developer and a sysadmin?

Unless you're a really small org, devs and sysadmins have different roles and need to have distinct privileges to prevent them from stepping on each other. The bigger you get, the more important this is.

You have a few things from an internal control POV that you can do to help prevent "bad" things from happening. Examples include: privilege separation, change control, dev/test/prod environments, scripted deployments, automation, etc.

When devs bitch about not having the privileges they need, the biggest issue is usually communication. The SAs work for a different Director, and there is an adversarial relationship. Fix that, and most of your problems go away.

Fun fact: Amazon and Microsoft have done "devops" since before the word was invented. Google and Microsoft do not. As usual, the correct answer is "don't hire idiots" and "do have clear policies, tools, and audits", not "don't allow person X to touch machine Y"

Devs and admins have different mindsets and attitudes, and that's a good thing.

Devs are more aggressive and forward-looking. Admins are more conservative and status-quo-oriented. You don't want folks with an aggressive mindset to admin your stuff. You don't want folks who worship the status quo to do development.

Too funny. I sure hope this is satire. Personally, I find writing chef recipes, building cookbooks, defining roles and managing the attributes of systems to be more of a beautiful art and craft than managing systems individually. To me, automating infrastructure feels more like a beautiful masterpiece than building one-off servers that are poorly documented and rarely configured the same.

It's an amazing thing to "knife vsphere vm clone" and "knife bootstrap" a system! Even though I built and implemented our automated architecture I'm still amazed every time I have a precise production system in under 5 minutes servicing client requests.

I think Chef may have found the equivalent to my systems administration G-Spot =P

I get the same feeling now that I've adopted TDD. I've lost that magic of spending most of my time debugging code.

Now I just define the outcome I want, assert that a unwritten function produces that outcome and code the outcome. Sigh /s

You forgot about interacting with third party code that you have no control over. Like when you develop any mobile app.

In this case you get the best of both worlds - if you mock the third-party API you can do TDD on all of your code, but you still get all the thrill and excitement of debugging when the API doesn't behave entirely as advertised!

They really are all crap. Run leaks - playhaven, tapjoy, flurry, etc are all guaranteed to be there. They dont even run static analyzer, which find some of these.

I was also talking about vendor libraries, like iOS UIKit and so on.

The title should be updated with a [satire] tag.

As to its content, I would say that server administration as a craft has moved up a level. It is now developing programs like Chef, designing 'clouds', choosing hardware and so on.

Two words: Artisanal Servers

I chuckled at the obvious satire. However, I have heard of 'sysadmins' that are complaining with less humour in their voices. The 'pure sysadmin' role is alive and strong. Those being forced out by devops are either not very good or the organization's culture is at odds with what a true sysadmin is expected to do for them.

Also, in before, "Lol, Windows admin". I cringed when I saw a registry reference.

For a long time I kept waiting to meet these "job security" sysadmins. The stereotypical 'if things worked better they wouldn't need me' guy. Haven't met him yet. Most admins I've met are automating everything that is reasonable. Many are automating MORE than is reasonable.

The only disturbing trend is how far the helpdesk types get in the hierarchy. My shock at introducing an IT Engineer to ubuntu, only to discover he'd never used a CLI on any system...

Any sysadmin going the route of 'job security' through obscurity(i.e., no automation or documentation) should be classified as criminally negligent.

My speciality is disaster planning and recovery. I consider myself being hit by a bus a potential disaster. That freak occurrence should not mean the end of a business.

And don't know those helpdesk types who have never touched a CLI. They allow me to bill a fortune when I step in to fix their botched deployments.

IT: adapt or die

> I too consider myself an artist and a craftsman of server building. With each click of a mouse, I create a work of art.

I... You're setting up servers with a mouse?

Haha, pretty much. He also referenced 'registry' tweaks. Windows admins...

I don't understand where this sysadmin is coming from. From a practical perspective, what's the point to drive the wedge between DevOps and 'traditional' sysadmin work. He's just critiquing the work of fellow administrators. They are BOTH sysadmins. Just as higher and higher level languages come out, higher and higher level abstractions of administration are required. With greater complexity of systems, more administration is necessary.

Deployment is so much more complicated today than it was just a few years ago. What we call 'DevOps' is really just a subset of systems administration work. Every developer (yes, sysadmins are developers) has specialties regardless of the term of the week.

Edit: Care for a reply with that downvote?

It wasn't my downvote, but I'll bet it's because the article is satire and your response didn't quite catch that. ;)

Building platforms is akin to building cars by hand - few do it that way anymore, and those who do are premium brands, built for very specific purposes wherein it doesn't make sense to automate those things. If cars were built all by hand, one by one, by "artists" they would not be affordable or as reliable as they are today. There is consistency, there is scalability, and there is reference in automation.

Do you think Amazon builds one compute node at a time? You may think it is art, but you can conversely look at a very sterile, well built environment where there is no room for error as art as well - something produced by DevOps.

Finally DevOps threatens nothing, at least nothing that shouldn't be. It promotes progress. Why? Because if your skill and craft is so good, why not be able to push it to the masses. And if you want to remain in the realm of niche then find the players that need one-off art. Past that, DevOps is long overdue.

DevOps hasn't ruined anything. Get over sysadmin as "art", otherwise I have unicorns and rainbows to sell you. Sure, there are good sysadmins and bad, but it's a stretch to say that you're so good your job can't be replicated with a build framework that fosters sane and repeatable process.

You've perfectly grasped the author's point, and repeated it in your own words.

So, that piece was satire. Unfortunately, satire is frequently used to attack the opposition without actually having to make a cogent argument.

> Get over sysadmin as "art" > Building platforms is akin to building cars by hand

Uhhh... no. My server is art. Now, I built my own build system, so do I really build servers by hand? No. But I built the build system. And I configured it. And it produces a server that is exactly to my specification. Down to every single file.

After the build? I click save. Then I duplicate. Then I have lots and lots of servers that are built to my precise desire. That's art.

Is my system more performant and secure than the system you built with:

(your favorite package manager) install (all the shit you need)

Yes. By a lot.

(chef or whatever off-the-shelf build system) install and configure (all the shit you need)

Yes. By a little.

But, hey, you know what? It's my server. And it's exactly the way I want it. And there's value in that. I own my stack.

Which makes writing software for it more fun. By a lot.

"Is my system more performant and secure than the system you built with:

(your favorite package manager) install (all the shit you need)

Yes. By a lot."

You should take a look at DevOps.

Reply on news.ycombinator.com