Confessions of a Full Stack DevOp | Hacker News Reader

Confessions of a Full Stack DevOp

ansible.com

115 pointsbrianshumate49 comments

Pet peeve: this URL is incorrect. When you're linking to an article on a blog, it's important to use the permalink for the specific entry (in this case it's http://www.ansible.com/blog/confessions-of-a-full-stack-devo... which admittedly wasn't easy to find - you had to click the heading on the article which lacks any clear visual way to tell that it's a link).

If you link to the blog homepage e.g. http://www.ansible.com/blog then in a few weeks time this post on Hacker News will no longer link to the correct content.

Thanks. We changed it.

We'll soon have a new way for users to suggest better urls, like you did, that is more reliable than a moderator happening to see a comment. I'll announce it as a Tell HN.

The last time I looked for a senior sysadmin -- less than a year ago -- I didn't get anyone who was comfortable programming in Perl/Python/Ruby until I started using the term DevOps. [1] via TFA

Ah, now I understand DevOps. Well, the sysadmin who can personally automate his job through programming--this is very productive person.

[1] https://news.ycombinator.com/item?id=7593681

In my personal opinion this is what DevOps is, in fact "OpsDev" might even be a better word for it in most cases.

It's about carrying out what are traditionally "Operations" tasks (Provisioning, Configuration, Orchestration, Monitoring, ect) using tools and workflows with are traditionally used by Developers (Version Control, Automated Testing, ect)

In large companies this work is often carried out by operations people who have expanded their skill set to include things previously limited to developer roles. In startups this work is often carried out by developers, as in the original rant, who are tasked with maintaining infrastructure as well as developing.

In the traditional development/operations divide, developers often have no access to and little insight into the production environment where their code will be run. As such, they cobble together whatever makes it work, and then "throw it over the wall" to operations, where it has to run in a very different environment. This makes for hellish manual deployments, done on multi-day phone calls with ops staff who don't understand how the code works talking to developers who don't understand the rules and procedures of ops.

If you see places where production code has to run in WebSphere or Weblogic because it's "enterprise", but want developers working in Tomcat to save money, run away.

This leads to situations where operations starts providing "developer resources" in terms of servers, dba support, maven repositories, etc, that turn into a tar pit where development gets stuck in incomprehensible bureaucracy, and "hack" around the system in order to actually get anything done. And then security or someone else in ops finds some little skunkworks dev system and has a fit because developers don't understand what they're playing with - or worse, they find out about the skunkworks because they get pulled in to fix what developers screwed up, with their own managers howling about who's gonna pay for it...

So to me, the beauty of DevOps is getting dev and ops working together in concert from the start to create a consistent development/production environment, where the same automation can be used to maintain both. This is a win for everyone.

And this is why DevOps has gained traction. Automated deployments, continuous integration, and software that functions identically whether in development env or production.

You can wax poetic about how it used to be, or complain about people being overworked all you want, but that's an issue of expectations. Personally I never want to go back to two week "integration" processes and multi-day manual deployments.

DevOps to me is an exercise in not making things more complciated than they have to be on the config/deploy side.

hmm, OpsDev sounds like sloppy dev who carelessly breaks stuff and says Oops!

Perhaps even more important is the ability to give developers, and managers, visibility into things which were only visible to System Admins in the past.

Why is this server running slowly? The graph shows a spike in disk IO usage. What's the process? The database. What changed at the spike? Adam's production push. Why? The explain plan for the new front page query shows a table scan against a 5m row table.

And done. No sysadmin needed, Adam can now identify and fix his own problems. The management can now watch the graphs of application response time go back down in response to Adam's fix, and everyone's happy.

This isn't theory, it's the example I saw demonstrated at a database convention a few weeks ago. Graphite, logship, nagios, and some development which pulls all of this data together can create some virtually magical systems. Casting devops as a side project for "real" developers limits what could be done to facilitate the creation of real agile development environments.

> And done. No sysadmin needed, Adam can now identify and fix his own problems.

> Graphite, logship, nagios, and some development which pulls all of this data together can create some virtually magical systems.

Who do you think pulls all that data together? DevOps/Sysadmins.

I build and wrap together the tools that make your job easier. And I take pride in that. I'm also on call so you don't have to be.

Relevant XKCD: https://xkcd.com/705/

> Who do you think pulls all that data together? DevOps/Sysadmins.

That's what I was trying to point out. A devops engineer is involved in the creation of these systems, however she does not need to be involved at 3am when the problem is discovered, since Adam can identify his own problem and can verify his fix did what was necessary.

> That's what I was trying to point out.

Ouch, my mistake. In that case, I misread your post, and I apologize.

The sysadmin who can not personally automate his job through programming is not a competent sysadmin.

In between putting out fires and occasionally setting up new systems, this is what sysadmins have been doing long before the word "devops" came into being. Anything you had to do twice, write a script for it.

At least that's what I remember from my sysadmin days: automate the shit out of everything so we could spend more time on Usenet, IRC and reading BOFH stories. This of course to counterbalance having to be on call to fix issues at 2am on a Saturday.

This is a really great point.

If you set up your systems right, they pretty much run themselves.

I've been doing systems contracting for a few years and contracting often turns out to be much better suited for this work than full-time employment (unless you're a company at significant scale).

My typical contract goes like this:

    - gain an understanding of their current systems
    - document *everything* that matters in their systems
    - simplify *everything* that matters in their systems
    - organize *everything* that matters in their systems
    - secure *everything* that matters in their systems
    - automate *everything* that matters in their systems
    - monitor *everything* that matters in their systems
    - train their engineering team on how to run the systems

That's it. If I've done my job right, they'll never need me again unless something exceptional happens.

The best sysadmins make themselves redundant as quickly as possible.

The best sysadmins put the power of the systems in the hands of the other engineers.

The best sysadmins are well rested because the systems they set up are so damn stable and boring.

How do you know you have a bad sysadmin?

    - they are a 'hero' that often rescues the company
    - no one understands the systems but them
    - they are a bottleneck for deploys
    - little is documented
    - little is automated
    - they work long hours
    - they're often frazzled and stressed

A big part of DevOps is saving developers from bad sysadmins. Fortunately, the last few years have been a renaissance for usability in systems tools (like Ansible for instance). This newfound usability for managing and automating systems has finally put the power of systems management within the reach of many more developers.

For more, see my posts on these topics:

Boring Systems Build Badass Businesses https://devopsu.com/blog/boring-systems-build-badass-busines...

Metrics that Matter for your Systems https://devopsu.com/blog/metrics-that-matter-for-systems/

This is a much better, more sensible article than yesterday. My good takeaway here is that DevOps means whatever you think it means - it's a lot of different things to a lot of different people. That's a much more positive spin on my frankly negative opinion that it's devolving into a meaningless buzzword to attach to resumes, recruiting, and PowerPoint presentations for management.

I know I came in to DevOps (the word, not the job - I've been doing the job since the mid-1990s) with a strong opinion of what it ought to mean - cross-org cooperation between development and operations. This article has me rethinking that a bit.

Yeah, the word is only meaningful when the speaker and listener have enough shared context to pin down the meaning. In that situation, there's probably a better word anyway.

It's not quite the red flag that "ninja" and "rockstar" have become, but it definitely makes me wary when I hear it.

Here's my unsolicited opinion:

Dehaan is pretty much spot on. Knupp's post seemed to first latch onto the idea that devops is about devs doing ops works. To add, I think Knupp should do a rewrite after listening to some devops cafe and food fight podcasts, if he really believes in what he wrote.

It's somewhat an unfair comparison though. If you're the author/creator of ansible you surely have spent a lot of time thinking and understanding what devops means. To be frank, I cannot tell if Knupps post is just link bait to sell his book.

The concept (and value) of the devops mindset & culture is less debatable in 2014 than in the past 5 years. Those who latch onto the totem pole model will simply get left behind while the rest of the industry moves on... or at least that's what they tell me!

Knupps idea of the totem pole rubbed me the wrong way too. I've met great developers who didn't have a clue about how to scale or monitor their software outside of their framework. How to properly deploy across DEV, UAT and PROD without making code modifications. How to keep these environments clean and flexible to enable these deployments. Nor should they have to. Call me a Dev because I read and write code. Call me Ops because I do deployments and fix production issues. At the end of the day I solve problems and just plain make stuff work. With or without so called DevOps tools.

You've met great developers who couldn't deploy code?

Sure. I've met great ones who could too. I don't think deployment is part of the development job description. Or is it?

For me, the idea of devops was always to help developers gain enough insight into operations, to be more useful. I don't care if you can easily test your war-file from within eclipse -- make sure that it can be sensibly deployed and upgraded on a locked down headless box.

Basically turning great developers into great system architects.

To me, it seems like it makes sense that a great developer should absolutely know how his code is running, how his changes interact with the production environment in which the code runs, etc.

You'd take a hit if you visit any big company that's not centered around IT.

It's quite common for the developers to be forbidden from knowing the details of the production environment, and usually they have no easy way to get that info. But don't mind, a DBA will think about the interaction between the several programs, and the ops people will think about performance.

You think that deploying code just means "push to master and reload/restart servers"?

Even if you think that it entails more stuff, like knowledge about AWS, rackspace APIs, provisioning tools and whatnot, it's not enough. Mostly because in any medium to large company they build tools and processes on top of them. Have fun learning all of that, while you only need to commit your code.

Plus, since I've been for many years a sys admin guy that turned into a developer, I can tell you that there are different skillsets involved, albeit with some overlapping ones.

That's where the devops come in.

No doubt deployment is not a simple process in many cases, and I have respect for the ops position. But the line I was referring to specifically stated, "I've met great developers who didn't have a clue about how to scale or monitor their software outside of their framework." To me, being completely clueless once you leave your IDE is a pretty strong sign of a "not-great" developer. For instance, how do you know how to write good scalable code if you have no clue how the sausage is being made? Not saying you have to be an expert at the full stack, but you should have an idea of the basics.

Nothing like finding urls (and even passwords) hardcoded in the source, huh? :)

I try to make any developer who will listen to reason at all read the Twelve Factor App manifesto.

Nice. Never seen that manifesto. Just pushed it out to all my devs.

It's worth taking to heart. It seems to have come out of Heroku-land, and it's baked into Heroku. Of course, that's an even more restrictive environment in some ways, but the underlying concepts are true for larger and more complex things than can be easily executed in Heroku.

Now all you have to do is get them to grok immutability...

Those don't sound like great developers to me. Those are the kind of people who are going to introduce subtle bugs and say 'welp, works on my machine closes ticket'

> When done correctly, automation tools are giving [developers] time back — and helping out of this problem of needing to wear many hats.

I've found this to be the case both in my full-time job (enterprise/healthcare industry) and in my larger side projects. Using some simple tools to make sure both server environments and deployments are uniform/repeatable/known in a good state has given me much more time to focus on actually developing.

Additionally, there are times when certain features (especially surrounding performance and scalability) requires a deeper knowledge of part of your stack. Being a good developer requires you to be a specialist (in your primary language/tools) and generalist (in almost all other areas) at the same time.

I would also add that you just don't automatize stuff. Because the more you add layers, the more you'll have to dig into at 3AM with the customer screaming at you on the phone. It's using the complete stack and using the least technology required to do the task at hand. While having access to the whole picture.

I have heard too many times that something is complicated then we can automatize it. But automatization has to be debugged, so, if the task is not complex per se, try to find a simpler tool, try to limit your intervention in the config files, try to limit the number of tools etc. But you can make those tradeoff across the whole stack. And suddenly you're balancing between adding a library to the code or a program on the machine, or a function package in the DB. You have a wider view, and you have the pressure to do the right thing because your phone is connected to new relic.

The "Let Them Eat Cake" antipattern, that is. Maybe no one has written it and I just made it up...

I don't get it. Google turned that up: http://robmyers.org/2009/09/21/free_software_debate_antipatt... and I don't get it either.

Basically, the idea is that if something is hard to automate, then automate it and then it won't be hard to automate. Peasants have no bread? Let them eat cake!

I've encountered this a lot in the enterprise. Baby-step automations aren't allowed because they don't contribute to a Grand Unified Theory of Automation that will Solve All Our Problems. Worse, you wind up babysitting antiquated automation that should be replaced, and you're not allowed to replace it because it's already there and already works.

This is pretty much a good article on DevOps. In my words i see DevOps as a bridge spanning the kernel, systems, networks, qa, packaging, and deployment. for a highly optimal system all these need to be planned for with care and attention. DevOps is a tough and long learning to be good at.

Great post. While I agree with this, "Automation shouldn't be about spending all your time with a tool ..." I wonder if employers are pressured to push you into more and more Ops work by this paradox? http://en.wikipedia.org/wiki/Jevons_paradox

This kind of situation, specifically (relative to other work): http://upload.wikimedia.org/wikipedia/commons/b/b8/JevonsPar...

Back when I thought I was going to be a Mechanical Engineer, I had a summer course that included surveying along with machine design and robotics. I felt the latter were more relevant to me, but had I gone on to work in oil fields in SE Asia I probably would have appreciated my time tromping around campus taking measurements.

I see all of this as just 'engineering' work, but it's easier to delegate certain things than others. When I ask someone new to configure Postgres for secure remote administration, script or otherwise automate backups, and rough out a schema for some tool we're thinking about, they'll likely accomplish at least a subset those tasks. The remainder is an opportunity for discussion. Someone else might just ask me to build something, and we'll talk about usability when he's done with the physics side of things.

But if I leave, what does that physicist look for in a new hire? The skillset of an 'engineer' is frustratingly broad and vague from a hiring point of view. Getting so-called DevOps out of the way might give another new hire the chance to develop as a programmer, and vice versa. Hopefully one or both of those two will eventually develop into an engineer.

I am a horseshoe crab. I am a sysadmin.

I can code, I do code, but I'm a much better sysadmin, mainly because I've been doing it as a dayjob. You want networking? I'll do that. you want multi gigabyte a second storage arrays? I'll do that. You want monitoring? I'll do that.

You want me to recompile python 3.3, naaa you can do that. I can, but you'll do better at it

Wow mate, don't get me wrong (I'm a SysAdmin/DevOps guy) but...

From where I came and from my background... RPM/DEB packaging is a SysAdmin job.

And... In a properly built infrastructure, what is recompile python 3.3 if not just repackage it?

because unless you work in a startup you have to support legacy, which involves recompiling python(or anything else for that matter) because someone previous did something stupid....

DevOps is really the traditional Unix admin role, however the operator role has now be renamed sysadmin in most companies.

"Full Stack DevOp", worst three words of 2013 in the same title.

2013?

It's just a euphemism for 'engineer'.

But you can't be an "engineer" without a Engineering degree from an accredited university, so DevOps.

That's true in some countries, like Canada (where the term "engineer" is trademarked), but it's not true in the U.S. Here, everyone is an engineer.

Unfortunately, the profession is leading the credentialing at this point. I'd expect that to change in the next decade or so, ideally resulting in a useful PE designation (in the US anyway).

But how can the sharing economy disrupt the cloud without full-stack agile devops?

I can't read this article. The font and spacing is horrible. Makes my vision shimmer and shake. Perhaps I'm just getting old.

DevOps and "full stack" and Agile and the flamewars around them are just bikeshedding. None of this gnashing of teeth actually matters.

You know what matters? Writing code and making it work. Solving problems. Helping people. If the right way to do that is for me to make a system more reliable, then I'll do my best even if I'm not interested in being "an ops guy". If the best thing for me to do is read machine learning papers, then I'll do that.

The real problem: technology companies have terrible leadership. You know how small towns in the Midwest used to send their genetically unfortunate to San Francisco, where there'd be more social services and a warmer climate? Well, that's what the business world does with its rejects, too. Buses 'em to San Francisco. Except, in tech, they end up making $400k as VPs of HR at startups and get to boss nerds around.

"Agile" won't fix that. Nor will "DevOps". These terms mean too many different things to different people, and ultimately fail to address the real, core issues with technology.

Reply on news.ycombinator.com