Why One Programmer Doesn’t Do DevOps Anymore

Its like the "Agile" push that companies thought they could get by buying a tool, and changing nothing else (release schedules, feature based release dates etc).

My favorite way of looking at it is simple - replace the word DevOps with empathy. That is the core of the idea, so lets try it - "Empathy Team", "Empathy Person", "Empathy Engineer".

DevOps is about dev and ops teams communicating and having empathy for the requirements of the other team.

A lot of us old sys admins are now calling ourselves devops because times are changing. To me devops isn't about taking a developer who likes ops to save some money. Its about taking ops out of their walled garden and into the development process. As someone in ops, I'd much rather be part of the development process as I'm the one supporting the code in production.

If its a choice of message queues.. I'd rather have a vote to the one I know about supporting.

It isn't really about cost savings as some companies I've been in I'm one of the highest paid engineers on the team. Its more about bringing ops into the process.

> It’s easier to train a developer to do sane operating procedures than to teach a sysadmin the developer mindset (testing, revision control, breaking down to logical units and similar).

Having done both jobs, I don't really think that's true, unless by 'sysadmin' one means the type of low-skill position which resets passwords. A system administrator is responsible for administering a system of hosts and software; his should be a more complex job than a developers. He should be responsible for test systems as well as for production systems; his scripts should live in source control. I'll grant that functional decomposition may not be as vitally important to his work as it is to a normal developers—but then, his scripts will inevitably grow to be pretty complex, so maybe it will be after all.

That doesn't mean that developers are stupid either. Sure, at BigEnterpriseCorp a developer is just a seat-warmer who implements directives like 'add a method to the Foo class which takes two arguments, adds them and returns the result,' but a real developer is a student (whether formally taught or informally self-taught) of computer science; he understands and can choose & implement both algorithms & data structures in order to solve a problem.

A good DevOps hire is going to have some skill in both areas, not as a weak sysadmin and a weak dev, but as a strong sysadmin and a strong dev. A good DevOps team is going to be more skilled (and expensive) than the equivalent dev and ops teams, but it will also be more productive and less error-prone.

If you're trying to build a devops team from Initech, Initron and Initrol low-skill employees, you're probably going to fail. If you're building a devops team to reduce costs, you're probably going to fail. But if you're building a devops team to increase profitability and competitiveness, you may succeed.

DevOps, when implemented correctly can have enormous, order of magnitude increases in both productivity and quality, but it is not voodoo, despite this almost unbelievable claim. It involves empathy and cooperation and it requires this because in order to implement DevOps in a meaningful way, an organization has to vastly increase communication between "silos". It has to formalize, document and automate as many processes as possible.

DevOps is NOT easy to implement and involves a cultural shift and absolute management buy-in. It is best implemented in small projects, where clear wins can be demonstrated. Use this success to bootstrap the process to bigger projects.

I do agree with the OP that if you have a DevOps team, you are probably not doing DevOps. The whole point of DevOps is to get rid of silo mentality. No more us and them, but all us.

Implemented correctly, everyone is DevOps.

I've also had more or less the exact same experience with DevOps, though with (hopefully) a happier ending. I was on a team working for a client that desperately needed "DevOps-type" things. So I built them, and their infrastructure team ignored them.

Eventually we stopped working with that client, and I was moved to a different organization, where each development team was doing it's own thing, operations-wise, and had no interest in adopting a uniform approach, so my (short) efforts there were also for naught.

But then I had the opportunity to come back to the first client, as their DevOps Manager. Before I took the job, I talked to their current development organization about what their pain points were, and what actions they would be willing to take to alleviate them, and I built a strategy around that. A key part of this strategy was gradually eliminating the need for a DevOps team, as I firmly believe the best application of "DevOps" is as a joint project between Development and Operations to optimize shared processes. Once the friction is minimal, you don't need a team there anymore. So we have a developer embedded in our meetings, and have more or less full control of any infrastructure.

1) cost.

Ops are cheaper than devops, sometimes twice as much. Yes yes, flexibility. But thats a misnomer. If your dev teams are doing donkey sysadmin work, because they don't have the experience, or are up all hours fixing things, you're going to endup with a high staff turn over.

2) division of labour

As a company grows, they can employ more and more specialist staff. This gives them a competative edge as they are able to make nuanced decisions based on skill, or experience. A decent storage admin will be able to build a storage system that will scale relaibly, cheaply and with less downtime than someone who is not a specialist.

3) burn out.

A human is not designed to multitask 14 hours a day. Coding, testing, supporting, infrastructure procuring, for anything other than the smallest company is too much for a single role.

There are great things that come from a devop "culture"; greater personal autonomy. However like all things, its not one size fits all. There is a great deal of arrogance where people assume that a sysadmin never automate anything. or that they can't program.

They can and do, you know why? because they want to carry on searching ebay, scamming $someone boasting about network speed, or just plain getting shitfaced.

When I work with clients the inevitable question comes up: "What Principles should we implement then?" and the answers are usually more questions really. How long does it take developers to set up their working environments and get to coding? How is code tested and prepared for deployment? Are stakeholders feeling secure about deploying this new code to production users? Can we deploy effectively every time without deployment related issues? When something bad happens in production, what is the recovery procedure and how long does it take?

The answers to these questions and many others often show that there are a few issues. First, developers and operations aren't speaking to one another enough and aren't really working together. Second, there is very little trust in the current environment and that lack of trust is made up for by adding more human intervention. Third, there is very little automation and this causes the lack of trust but few people what to change their habits to fix it. And finally, there is such a great divid between engineering and operations that the pains felt by one group and not shared by the other.

DevOps has the principles needed to start to solve these problems. However, we don't need a department to implement those principles. We have enough departments.

Dedicated QA teams should be avoided unless truly necessary. Otherwise, you have the same problem of developers throwing over under-tested code to a group of separate people that are smart but treated poorly.

A QA guy who loves to write automation gets the same promises... "oh you'll be programming mostly; manually test maybe once a month". But when they get into the door, it's the exact opposite.

Developers should all be passionate about QA (read as: the quality of their product), as well as how their product is actually deployed (read as: executed in the most important of environments--production!).

'Do not set out to create a DevOps team or employ people for that role'

My interpretation was the role simply doesn't exist. If your company has a team of developers and a team of people who keep the wheels turning - in an ideal world you already have a DevOps team as long as they collaborate. If you also employ smart people you will get 'developers that know how to do sysadmn' and 'sysadmins that know how to develop' - definitely this indeed should be embraced but not expected.

"Devops means giving a shit about your job enough to not pass the buck. Devops means giving a shit about your job enough to want to learn all the parts and not just your little world.

Developers need to understand infrastructure. Operations people need to understand code. People need to fucking work with each other and not just occupy space next to each other."

http://blog.lusis.org/blog/2013/06/04/devops-the-title-match...

I cannot disagree more. You can't have developers of your product knee deep in DevOps and be effective (at a certain size, smaller teams with less demanding operations needs can).

DevOps should have very clear demarcation of responsibility in an organization. What the OP describes is an blending of responsibilities and that is the root of his problems.

DevOps /is/ operations. It is just that simple. It is design and automation of support systems. Nothing more, nothing less.

I say this because I've lived it...the last 3 (three!) companies I've contracted at were pushing dev code to live servers! Even after I setup svnserve and created the repos, no one was really interested in changing workflow. I choose svn because, lets face it, it has a much lower learning curve then git.

It wasn't until I absolutely refused to work on the live server that other the devs grudgingly started using SVN...I'm sure they are all back to FTPing to the live server now that I'm gone.

Without the proper culture from above, things just aren't going to change.

My job, just like it was when I was a sysadmin, is to keep things running smoothly. To optimize both sides of the coin and to take things off their plate that I myself can deal with that might be a blocker to them finishing tasks and to us getting features out or server migrations done.

Some devs know ops and some ops know dev, those guys, more often than not I just check in with and see that things are running smoothly. For the other folk I try to create systems and workflows that get blockers out of their way.

I try to be a useful and reliable tool to both sides as much as I can be.

Hopefully the way I've been doing this for the last two years since I moved into a more Devops related role has been helpful to everyone.

In theory that will eliminate the need for an additional tester and a second individual to do the change control.

But those short term gains expose the organization to the massive risks that the one and only one individual will make a mistake, with no second individual available to do a double check.

The reality is people make mistakes and by eliminating people from that double checking process may well save money in the short term, but in the long run it greatly adds to the of risk failure.

While there are short term gains, the risks to the organization can be massive:

http://www.bloomberg.com/bw/articles/2012-08-02/knight-shows...

My definition of devops: small teams totally own their project, iterating on design, implementation, deployment. I have no experience with one giant devops team for an entire company.

My experience is different: DevOps saves me many headaches by letting me easily deploy things for small clients. A larger company I work with uses Chef to manage their 40+ servers; the IT department doesn't touch these servers. Rather, 3 or 4 developers share in the work of writing cookbooks, adding instrumentation, deploying code and maintaining the servers. It would be handy to have a full-time guy as well (OP mentioned the amount of work needed to keep up with just your basic security patches) but we -- as developers -- love the access we have to our entire stack.

My take is that the code quality of whats allowed to be in production has gotten worse, that's it. It's an effect of an abundance of RAM and virtualization. The abundance of ram lets you just trash it and do things in a horribly inefficient way. Sure the extra memory can help with time memory trade off if you have something that doesnt need to scale a great deal, but processing things inline will still be faster. Developers have forgotten how to handle streams. Usually forcing a dev to learn a highly functional language that's immutable can fix these issues (it must involve a large dataset, where you know they will tax out the memory, and then code review to help).

I mention virtualization because a common pattern I've seen is to write poor code, and just put more boxes behind the load balancer when things start failing because the technical debt is so high it's an impossible mountain to climb. This obviously isn't the right approach and will leave your devops guys worn out chasing your everyday distributed systems problems that happen because it wasn't built right for distribution and the dev has never read the fallacies of distributed computing.

And while you're fixing this technical debt a team of competent engineers builds a competing product and trumps you hard (second mover advantage).

Maybe people paid a bit more attention when things cost 10x as much as they do now and every answer (both bad and good, and you can't tell the difference) wasn't a stackoverflow search away.....

Although in his post, he says:

"Another lesson learned: you can’t force cultural change. It has to start from the bottom up, and it needs breathing room to grow."

I would argue that cultural change has to also start from the top. You can't change the culture if the leaders of the company don't work with you on it.

The tendency of people at the bottom is to eventually start behaving like their managers as the managers project what is acceptable and not acceptable behavior. Trying to start change from the bottom up is a hard, hard thing to do, with a low probability of success if senior management isn't on board as well.

I was part of a successful transition to devops from a thoroughly silo'd operation, where the boss basically gave the dev team the responsibility and authority to make it happen. He was also able to snag some of the traditional ops team and make them part of the new team. After a period of storming, the new team came up with some innovative ways to get the job done. Once the team realized that they were it in terms of getting it done, it got done. Quality and job satisfaction improved, cost was reduced.

I've also seen various unsuccessful transitions. In every case, the responsibility was divided into various silos, which sandbagged everything.

The memory bandwidth in one single person's brain is simply much higher than the bandwidth between people. To maximize performance you want to run your project on as few brains as possible.

I used to be able to use "senior systems administrator" as a short-hand for those skill sets, but about 2010 that stopped working.

Now I need to use "DevOps" as an imperfect substitute in the title; otherwise only junior people apply, even though the rest of the ad goes into the same depth about skills and abilities.

What you want to look out for is the formation of "walls." People have an instinct to do this. People will form cliques -- which effectively are invisible barriers to understanding and honest collaboration.

In a 5 person startup, this is easy. You have one group. Most of the critical information flows are inside this boundary, and those which aren't, the group is already intensely aware of. (Investors, customers.)

However, when headcount has started to grow, and when you are forming a separate group to take on a task that requires intense collaboration, maybe you're doing the wrong thing? Maybe what you want is a separate role for people who get embedded in a variety of different groups? What if DevOps people belonged to DevOps, but also, and primarily, belonged with a particular group of developers? Then, they could have firsthand experience of intense collaboration, but also meet once a week with all the other DevOps role holders and communicate to standardize and avoid duplicated effort?

A generalization of this: Make sure you're building bridges and not compartments.

Sure, there's this general idea that development and operations teams should work together, or of a programmer also having some experience adminning systems or vice versa. But this is as much a job description as "non-domestic car mechanic" is a job description. Generalized ideas that don't say anything but an indication of the background expected of the person.

"DevOps" or not makes no difference; all corporations face the same problems with multiple segregated teams and people doing whatever they want in order to get their jobs done, as well as a sort of tunnel vision that prevents one from properly supporting your teammates or a product. To this end, there is only one thing that will make a significant difference in the work done, no matter what you call your team or position.

The best engineer in the world becomes a janitor under the wrong leadership or teamwork.

While I agree that sysadmins should be developers first it is increasingly less likely you'll find developers willing to become computer janitors and deal with the ugly realities of operations in, for example, larger companies. I wouldn't think that John Carmack ever liked having to figure out the installer scripts and keep them working across all the supported platforms through all the BS that Microsoft and different Linux distributions could throw at you, yet that's what a more "ideal" DevOps organization described by many model would have John Carmack do.

What should have happened though is that instead of hobbling the John Carmacks and dragging down their productivity, you should have fired the developers that refuse to cooperate with operations and those that keep actively putting off writing "boring" code or never documenting things that do matter for operations to figure out and learn through a failure that something broke. That kind of behavior is the developer equivalent of not being a team player at all. It's true that none of these things actually make you a better programmer in itself, but these things are what contribute to making you more PROFESSIONAL instead of a cowboy renegade programmer that doesn't care if nobody else can use his code because "clearly, if you can't figure it out quickly you don't deserve to use it" (not an exaggeration at all I've heard).

Having been in both development and operations in rather large (and small!) companies now, I'd far, far, far rather stay as a hermit-crab developer than deal with the constant dysfunctions of enterprise Faustian nightmares that are required for you to deal with as an operations engineer. You can generally ignore a lot of the noise in a super-siloed off R&D organization, for example, but you cannot ignore your pager going off at 2 am because a random server crashed causing someone to get very, very angry and demand immediate resolution when it is your JOB to keep those things going. Hardly going to write anything really meaningful with that ruining your work week pretty much every week.

The exact same thing I'm noticing is happening with the "automation" teams that are cropping up. It's like folks are discovering ways of getting around process, forgetting completely why the process exists in the first place.

Many of the lessons learned here aren't exactly universal, for example some of the better Devops cultures (e.g Google) do have separate dev & ops teams, they just have good collaboration.

> We had edicts to make things happen, under the assumption that if we built it, the developers would embrace it.

Followed by:

> What matters is good people working together without artificial boundaries.

Former will never produce latter. Always work with engineering before building anything that impacts their day to day.