How 'DevOps' is Killing the Developer (2014)
jeffknupp.com
jeffknupp.com
Being able to do something from memory is something we only care about in high school. Take me, for example: did my router support L2TP termination? No. Did DD-WRT support it out of the box? No. What a developer can do is cobble together information in order to find a working solution. So would I, as a non-Dev-Ops be able to set up MPLS over IPSEC, having never done it before? Yes: I have Google and I have a brain between my ears.
I think what @scurvy was driving at is there is a lot more subtlety to network ops than developers would be willing to acknowledge.
While just an example, it shows that our professions all have subtlety and nuance to deal with. You gain the ability to handle this through experience, not Google. Throwing a "full stack" developer against every problem is definitely not the solution.
It is a much smaller issue if you are building a system from scratch and only the launch date depends on your speed, but if you are trying to modify a part of mission-critical running deployment that already generates a revenue stream... well you have to first negotiate reasonable maintenance windows and then must squeeze yourself in. Googling around will give you zero clue how much time fixing broken things will take, because, well, deboogling is done by error messages as keywords.
Furthermore, only an experienced (I'm refraining from a term "trained" here) professional can evaluate multiple possible setups, give proposals to decision makers with more or less quantitatively expressed installation/maintenance/growth costs, evaluated downtime/security risks and business impact (solution A might forget the order while still charging the customer with 0.001% chance, while solution B might forget to charge the customer at all with 0.002% chance when the load reaches 47 requests/s, so which one?). Sometimes you just want to deploy something small quick and then at some time transition to something bigger. And knowing when that is reasonable and when the small system will stop handling the load allows to actually forecast and take weighted decisions.
To the defence of DevOps: cobbled together solution is often the best (weighing in deployment time/costs) for small internal services, first dev environments or early startup prototypes. And for large deployments, someone having a clue about both sides (code and operations) is necessary to moderate the decision taking process. Because what an operator thinks to be a quick change to code might actually take several man-weeks (who was the idiot behind this architecture?) and what a developer thinks to be a small change in configs, might actually require changing core hardware (what a moron deploys like that?). In my own short experience (including, but not limited to IT/CS/EE), I have failed to witness a project where everything is decided upfront and does not change until at least first (alpha/beta/customer preview) release. The more startupy the project (the less domain knowledge), the more first release target deviates from initial goal.
Just my 0.02$
More like $2000. Very enlightening. I was reacting more to the conclusive nature of this:
> The author says that developers can do these things, but I highly doubt it.
That is not conclusively true in all situations. If my employer came to me and said: "figure it out," I definitely could and would. I would first recommend they hire an experienced devops but it's not like I could never learn it.
Dev and devops are not different like brain surgery and heart surgery are. Either role can grow into the other role and eventually (for the most part) do well at it.
But really, the author here is just misguided. Just because you can 'figure it out' eventually, doesn't mean you'll be cost effective at it. Employee productivity comes from specialization. It's not about how many jobs you can do, it's how quickly you can do them well. When developers are paid well, it's because they have specialized knowledge, just like DBAs and network engineers. There's not even really a need to put developers on some fucking pedestal make the argument against 'full stack developers.' It functions purely as bait for angry comments.
I couldn't find this in the article.
You sound even more outraged and polarized than OP is. I've met more than one developer able to do this (hint, I can also do it). Some people are actually paid to write networking software, and they often have a clear understanding of how JunOS and IOS work.
This said, for different reasons, I agree that network operations should not generally be done by your development team, but no one is arguing this anywhere.
As the business grows, you're going to need some specialization skills for some project or deal that comes your way. You should hire the right person for that job. Chances are, you needed them all along and this situation is a good catalyst to get them in.
1) The developer would just say - no, instead I'm not going to use IPSec, and I'm going to bootstrap this myself with TLS. I'm going to push complexity UP the stack, not requiring esoteric network hardware to do this.
2) I think this is a bug. The fact that the underlying devices don't have easy to understand semantics is problematic. I think that if an interested developer had a program that requires MPLSoIPSEC, they would consider it, read the relevant RFCs, and then read Junos's documentation and give up - because it's locked down.
2) Again, you could use Linux for this. You can ask the whitebox switch guys, but there's a lot of stuff Linux can't do. It's a great cause and I've got a few whitebox switches in my lab, but there's definitely still a place for Cisco and Juniper in this world.
Where I work I'll push into production a few times a day after testing in staging and the only changes really are what container is running what code.
The firewalls and routing tables I don't manage and with good reason (unless they are on AWS since security groups are pretty cut and dry but even those basically don't change much after initial configuration)
However, the problem seemed to stem from cryptic tooling and almost non-existant documentation (especially on OSS tools).
OP seems to think "full stack" developers no longer devote time to writing code... "This is why we see so many developers that can't pass FizzBuzz"... but really it just doesn't take us long to do the non developer stuff because it's just easy these days, maybe he's not aware of the tools people use.
> "...release coordinators and the like are at the bottom of the totem pole. Why is it arranged like this? Because each role can do the job of all roles below it if necessary."
I find this to be belittling those who choose to work on a different part of the stack than the author. It seems to suggest there is an innate incompetence that keeps release engineers from functioning as "developers". That couldn't be further from the truth.
Release Engineering @ Mozilla: https://www.youtube.com/watch?v=7j0NDGJVROI
FUNNY STORY: we make the developers act as release coordinators, because it distracts us from the work of improving the release process by replacing it with automated software systems.
The top of his hierarchy should be PMs/executive team; betrayed by his own developer ego…
Sounds like some people need to try the "less technical" parts of the stack like requirements gathering to appreciate their roles.
It is the case, that in many organisations, the roles down the bottom of his totem pole are less skilled than the top.
There are point-and-click DBAs and Server Admins and release managers who don't have the base skills to write software. But they're rarely very good at the roles they're doing.
I have a bunch of issues with what a lot of organisations do under the banner of "DevOps", but part of what it is trying to do is bring engineering solutions to those layers "down the totem pole".
I want DBAs that invest in reliable, repeatable, automated processes. And that means I want them to be able to build working software. And similarly with the other roles he mentions.
Now, generally speaking, you'll get more useful work from a system admin who just follows the procedure they were given, than you will for a developer who does likewise. But, if you can do it, you will get plenty of value from hiring people with some engineering skills/training into all those roles.
The counter examples are the system admin who takes 2 weeks to setup a new server because it's all done by hand every time. Or the release team who buys fancy "automation software" to automate a broken process because they don't understand the process well enough to build a working solution. Or the operations team that just reboots the server when it's broken because they don't have access to people with skills to diagnose the problem. Or the QA team that builds automated testing tools that are completely unmaintainable because no one in the team has a background in software engineering.
The totem pole hurts because
- it (often) treats the people at the bottom quite poorly.
- as a consequence it discourages good people from taking roles "at the bottom".
- which leads those teams to produce far-from-optimal solutions. Not (necessarily) for lack of effort, just because they don't have the skills + training to build the optimal solutions.
(Obviously, as your Mozilla example shows, some organisations manage to escape that trap)
If you need to hire less qualified/skilled people, then you're probably better off putting them into the roles that are "at the bottom" of this supposed totem pole, but if you can fill all the roles with solid engineers then you'll reap rewards from it.
The article is inflammatory to a degree, but it does make sense applied to groups rather to individuals. Devs do have to learn a lot of the skills the other groups do, but it doesn't go the other way. There are still obviously valuable skills that a DBA or sysadmin will have and your average dev won't. It doesn't change the fact that, as groups, I'd rather have a group of devs be forced into DBA and Sysadmin tasks than the other way around.
Yet, they don't get that it's all about groove, rhythm and dependability and are just way too unrelaxed and attention seeking for doing it right.
I'll take a grumpy BOFH that wires together my AWS regions in his sleep and doesn't make costly mistakes - because he measures twice and cuts once - over some kid trying DevOps any day :)
Only the sysadmins "got it" because they were the ones that got to feel the pain of cleaning up a NAS filled up by log files.
Presumably logrotate was already set up for the variety of other logs on the system?
Developers want features and shiny things and following new trends and unicorns.
SysAdmins want 99.999%, and dependability and knowing that all the backups are running on time and writings contingency plans detailing how many minutes it will take to put a new infrastructure online in the event a fire occurring in the main data-center while the backup data-center is flooded because of a typhoon.
Of course they are not going to be experts at it, because their field is development.
You can be deep on a single area (specialist) or be spread out (generalist). Larger businesses want specialists, small business want generalists.
Personally I am a generalist. I deal with security, servers, routers, switches, applications, laptops, Operating Systems, Backup, Databases, etc. I write code in python, java and starting to get better with go. Its a handful and I am always juggling things, but I also think that broad knowledge helps things. I understand TCP/IP because I wrote a TCP/IP stack. I feel I have a better understanding of the complex systems as I have a non trivial understanding of each part.
Its a very different working situation, but it has its advantages and I dread ever going back to a large corporate because of it. It isn't as efficient compared to having a team of specialists, where everyone can spend 100% on their speciality, but on the other hand, you can often do things with surprisingly small teams.
Teams do DevOps. It isn't supposed to be an excuse to make developers do everything. It's supposed to combine operations and development on the same team.
This is just like the perversion of agility in development by "Agile."
This, too, shall pass.
If the premise of the article is "Developer time is precious, don't make them do grunt work" the I would counter with "Developer momentum is precious, don't let developers get blocked by operations/QA/release."
On our team, we had Ops and QA. But I needed something done and they were already working on something then I could have a quick 15 minute chat after standup about the Puppet change I needed to make and then go do it myself. An hour later I'm done, the change is deployed and I'm moving on to the next task. Now, without an embedded Ops resource on the team, I have to file a ticket and pray someone picks it up within a few days.
If you've read the Phoenix Project, it clearly states DevOps solves the problem of bottlenecks, which is the single most important aspect in the Product Development lifecycle. You just can't have developers waiting anymore for Ops to do something, or vice versa... Shit needs to go fast and blocking factors need to disappear, or else you sacrifice agility
Disclaimer: I've been a (in no particular order): industrial embedded developer, sysadmin, engineering manager, founder, security researcher, consultant and SRE.
While I understand the point.. I'm also a dev and I love "DevOps". Aren't we all doing this because we love problem solving? How is this not just another problem to tackle? I love having to figure how to automate mundane tasks (again.. this is just code.. not much different than we write elsewhere) to make sure I don't have to manage any systems manually.
/rant
And that CAN be a problem. It isn't necessarily always a problem. But think about it. How many of us prefer jumping up to try working on a brand new project. Shiny stuff everywhere. And then everything gets abandoned at the grind and polish stage. The enthusiasm is gone. We crave the next new problem that we can use as an excuse for jumping away from the grind we are in. I think what the author is trying to get across is that this new "devops" culture is creating a large number of developers who don't have deep knowledge of anything. Ask a developer (myself included) to evaluate a set of JS frameworks for a particular project which has somewhat specific requirements. Behind most answers (this is based on observation, myself included again) the primary argument will be the developer's preference; not an objective breakdown of what is most suited for the task. We might not see too much of this on HN (I think. I see a lot of intelligent debates here which I learn from all the time) but on a larger scale developers are becoming half baked.
And that's basically the other side of what you pointed out. Problem solving, learning new things is a lot of fun (discovering and configuring vagrant and docker for the first time was incredibly fun). But it possible that the full stack role is churning out people with broad but just somewhat in depth knowledge. Tough call. Question is, is somewhat deep deep enough for most everything?
I dunno about that. My experience has been more that a team may need one person who's both wide and deep (T-shaped, or maybe just "rectangular"), but the companies I run across generally seem to hire younger developers who do have one or two core competencies--I-shaped, if not just dot-shaped and hoping to train them up.
Though on the other end of things.. I have a friend that is very into how some languages are optimized for various tasks and he'll mention to me that X language does something with their loops for speed, etc.
I don't have that same low level knowledge of the languages (not as low-level at least; I have some) but at the end of the day X language is slower than those that I prefer for the type of work we do (APIs mostly) so I don't care that X language does all these things internally.. and I don't need to know because it doesn't matter to us (nor our employer) in the end.
So my--hate to use a buzz phrase here but it's appropriate--'get shit done' style tends to work for me and our company because evaluating tools comes down to "does a framework or tool accomplish what we need without too much overhead".
Is this the type of deep knowledge you were talking about?
1. It's really just a dumb buzzword. 2. I really don't like companies dumping more work on devs for the same pay and using a buzzword to justify it. Oh here, you get a pager now because we're DEVOPS! Hire a god damn support staff.
That said, letting people form little kingdoms and toss up walls so that people just end up getting bound up in bureaucracy is also terrible.
I disagree. If a component is essentially a black box, where I know the inputs and I can test to check I get the expected output in known conditions, then that's enough. It has to be, otherwise there wouldn't be any way to use a middleware, or a third-party API, or anything too complex for one person to understand.
There are lots of software businesses where no one fully understands the internals of the product. That's fine. So long as there are people who expertly understand the part they're responsible for, and the APIs are well documented, everything will work great.
The problem with devops is that it gives people the impression that everyone can be responsible for everything, with everyone on the team having a hand in the design, the code, the testing, the deployment, etc. For a sufficiently complex product that is just not possible. At some point it gets too big, and people have to start accepting that they can't do it all, and that they'll have to trust their colleagues to get things right.
If you know how to go deeper then you run a packet capture and start seeing that, whatdya know, your top-3 IaaS provider's network is shit and is often dropping packets, so those timeouts are actually occurring, though not on anything you control. Now you're able to try to setup a repro, then open a ticket with enough details to get something done. Versus opening a ticket along the lines of "Hi Top-3 IaaS provider. Some monitoring service says our site doesn't always respond, but I'm sure it's not me. Please fix your network, thanks."
Not knowing how things work at a deeper level leads to experienced teams working on high-profile sites to come up with stuff like: "We have 2 domains in use, but need to go SSL only. We only have a certificate for one domain. So please implement a DNS-level redirect from domain A to domain B to avoid browser SSL errors."
Or an experienced dev insisting that a major vendor is shipping invalid binaries because the runtime says so. And insists they aren't metadata-only libs, but isn't comfortable opening up the binaries to verify and thus spends a day trying to figure things out another way.
Being able to get to a baseline of what's working is a fundamental step in troubleshooting. Just like you would always ask a home user to make sure stuff is plugged in, verifying lower-levels of the stack are correctly operating should be a common step - it's unfortunate how common it is that something else is broken under the level you're working at.
In the case you're alluding to, it'd be much more useful to know general information about why any monitoring service might think any IaaS provider is not responding, and have the skills to test that. Knowing details about the components you're using (eg AWS and Pingdom) is much less useful.
When developers are in charge of delivering a service, they will end up spending most, if not all, of their time debugging operational issues instead of improving the service. This problem gets worse over time as the service will need to be deployed to more places to address scale.
An organization with good technical leadership should know this.
There are two recent trends I really hate:
DevOps and the notion of the "full-stack" developer
But our company goes the other way, they make Devs do Ops stuff and QA stuff (and bussiness analytic stuff also). I really hate this.
That said, developer/unit testing is one aspect of QA you just can't offload to a separate team.
"""This is why we see so many developers that can't pass FizzBuzz: they never really had to write any code."""
Doesn't really follow from the previous parts. DevOps is not the reason for this nor is the full stack developer trend.
The author also seems to start from an implied base premise that developers wouldn't do devops/fullstack if they had a choice. I disagree with that. Many developers are very interested in understanding the implications of their code better (more DBA skills help a ton for example) and quite a few gasp actually enjoy devops or understanding the entire product from start to finish at least on some level (full stack).
The benefits of DevOps also seem to be completely ignored. It's precisely the fact that common development best practices are integrated with sysadmin, dba etc. knowhow which makes it useful. It's not just "let the dev be a sysadmin in a pinch". It's systematically improve things by automating them (following development best practices instead of the hacked together bash script approach).
DevOps and fullstack dev are conflated throughout the entire article in a strange way which makes it kind of hard to argue.
DevOp makes sense to me for building complex solutions..but, for different reasons than discussed here.
Consider this: a small team (usually 3-7 members) of "jack of all trades" is proven across industries where the team has to deliver a complex solutions under a very aggressive timeline ,requiring multiple skills. Two examples from my personal experiences: Strategy consulting - eg @ McKinsey (work includes interviewing executives, presentation, excel, data entry, hypotheses development and verification....) investment banking eg @ Goldman Sachs (work includes industry analysis, financial analysis, scenarios, presentation, excel, data entry, ...)
"Jack of all trade" model in terms of DevOps works because:
1) goal for the team w/ "full-stack" skills is to have a reliably working software solution 2) communication and management overhead increase significantly as the team size grows 3) complex solutions need different "amount" of various skills at different point in solution cycle 4) adding/ removing team members comes w/ significant cost to alignment
The way I view DevOps is that you have end-to-end guarantees about software delivery. A lot of this draws from personal experience, so YMMV.
Story:
I was working in an organization where engineering had built a custom solution with their own database to a problem. It was using their own DB system, with the JVM, etc.. I was oncall for said service, because their DB kept getting into a poor state. My only real recourse to a page was (a) escalate to the devs (b) restart the service. I ended up furiously rewriting the service in Erlang over a weekend, and ended up getting told that my replacement was too complicated, and the previous one worked okay, because we could just restart it.
Take-aways: Without feeling the pressure, and side-effects from being woken up, and having to respond to on-call incidents for a service, it makes people put more value on the development time than operational time, although a service is very likely to be in operational time much longer than development time.
I have two other stories, but it'll take some time to write, and it's getting late, so I'll type it up only if someone is interested. Just so I remember - the tale of LD_PRELOAD and HDFS, as well as the networks and root access.
1. I think that ops staff should provide shared services to services organizations (hosting, deployment, etc..) when N services need the shared service (when N>2, depending on how many resources are available in your org).
2. I also think that ops staff should be able to "consult" on projects, as members of those teams during development. This is going to mean that those engineers are probably going to need to do some more development! Or, alternatively, the project team needs to be able to backpedal on some decisions if those ops consultants come in later on the project.
3. I think dev folks need to have end-to-end responsibility, including oncall, AARs, and on-call remediation. They need to take feedback from ops folks and act upon it, just as they would if product told them to do something.