The Developer is Dead, Long Live the Developer
paperplanes.de
paperplanes.de
So while people will probably dedicate 80% of a week to a given kind of task, they can, and do, play multiple roles, depending on how much their specific expertise is needed that week.
For instance, I am the number one choice on UIs, but I also play support managing Postgres, because I did a whole lot of that in a past life, and we don't need a full time DBA. I am also leveling up in our scheduling system, all Akka actors.
At my previous job, the team had all the Database experience it needed, but domain knowledge was weak, so I spent much of my work working with users trying to figure out what they needed, and why they were unhappy with the product. They hired me for my UI strength, but reality said that I would be more useful doing customer facing activities half the time, so I did.
We are moving into weird heterogeneous bugs territory. A lot of stuff that has wasted my time last year was caused by interaction of technology stacks and not by pure logical errors. Or error localized in even one package.
I had problem with premature socket close on the response with just the wrong setup for all of nginx, haproxy, slim and php 5.4 ... lets pure ops or pure dev figure that out.
I rely on the ops guys to keep vanilla installs steady and to keep everything patched, but there's very little way that they can get insight into application problems (especially in the framework du jour) that express themselves at some other location in the stack. I just have to know the whole thing.
I on the other hand have no admin rights (only read access) but I know enough of out network and ops structure to be able to make an educated guess when something is behaving strange.
So if it hits the fan they could probably patch an emergency bug and I could fire up some VM instance. It is not about wearing a lot of hats, but being able quickly to put on a new one if needed.
As in all things in life -- the truth is somewhere in the middle. If you build significantly sophisticated systems you will need specialists at some point. I don't recommend wasting their talents on getting reamed out for forgetting to configure the firewall properly.
We don't hand off code to be installed, we give a release plan, an installation plan, and the code. We are on call for first runs, and there's always a subject matter expert developer on call to support overnight issues if the support team needs help. If you have this responsibility, you write more responsible code. If I may support something overnight, I'm not going to unthoughtfully make it hard to support. We share testing and documentation responsibility as well (although documentation is an issue), so again testers and support are doing their job in a vacuum because they don't have the developer's insight to the program. In fact, devs do most of the testing. It's not always easy to write test cases for a system you didn't write or design. I've kind of always thought that it's common sense to support your own product in test/production as needed (though not the first point of contact). You wrote it, you're the expert.
Reading this article surprised me because I thought it was the same everywhere. Having no responsibility for testing or prod seems like asking for bad code. If you're just writing code, you're not going to focus on what it's going to be like to support it, install it, maintain it. You'll have other, more concrete (to you) things to focus on. Coding standards may become more important. But are standards really as effective if they're not informed by / don't care about responsibility for a production product?
I think it's perfectly reasonable for a developer to be aware of excessive memory fragmentation, leaking FD's, and anything else related to the process their program directs. It's well within my domain to debug such issues.
However the "devops" role is also pushing responsibility for configuring firewalls, LDAP ACLs, securing the OS, managing init daemons, and the entirety of the infrastructure on developers. Since it's all code now there can be more people involved and the silos can come down. "You build it, you run it," as you say.
What I'm suggesting is that for significantly sophisticated applications it won't be tenable to spread the responsibility on everyone. It's not reasonable to suggest that everyone be familiar with the implementation of a simulated-annealing algorithm in the continuous-domain constraint search solver. I think a support person would be well within reason to defer questions, bugs, and requests with that particular part to the developer responsible for it. And yet we're arguing that it's perfectly acceptable for that developer to be familiar with the intricacies of managing a data center of compute clusters? I don't know about you but I don't have an infinite amount of time on this Earth to take on such responsibilities.
I do think that there's a middle-ground to be exploited. If our expert system is ballooning in memory usage and crashing our servers it's reasonable for the developer to profile the memory usage under simulated loads and data sets and look for memory fragmentation or leaks, etc. I think it's a little too much to ask one person to also be intimately familiar with managing compute clusters, RAID failures, security policies, and package management. They, like many of us, may have a good, general idea of how it works but there exist specialists for a reason: it's hard to go beyond a certain degree of complexity without focusing. It's the reason why we work in teams!
No one is arguing for that, as I said. Where in the article do you see something like that? No mention of firewalls, data centers, developers doing their own server administration. This seems to be a more important takeaway from the article:
> Ultimately, DevOps is about empathy, with everyone on your team and with your customers.
No one is suggesting "everyone does everything" at all... you're sort of creating an argument here.
Regardless, this pretty much only applies for SaaS shops I'd say. There's still software delivered the old way that's viable.
However I have also seen this branded as "soft skills". Something that seems to be yet another way to excuse many people for their lack of skills and reward them for doing other stuff. I have seen it used to keep really bad developers, if not reward them. It also tends to be a favorite term of those who cannot manage their team properly.
1) Unit test. 95% of your test suite should be unit tests. Objects under test should not require the entire codebase to be loaded in order to perform... ideally they depend on nothing, or just 1 or 2 things (which in turn hopefully depend on nothing, or just 1 or 2 things, ad nauseum).
2) Don't mutate passed-in arguments. In fact, mutate as little as possible.
3) Don't use objects with potentially unexpected, surprising or unknown behavior that makes it difficult to reason around the code (QUICK- what happens when you merge a ruby HashWithIndifferentAccess with a regular hash containing both similarly-named symbol and string keys?)
4) Use function objects, that have zero side effects, wherever it makes sense. (Or just use an entirely functional language.)
5) Separate components that talk to each other through I/O should use something like the circuit-breaker pattern http://martinfowler.com/bliki/CircuitBreaker.html
6) Microbenchmark things. Individually, a series of tasks might all "look" fast, but when run 10000 times in a row, might expose unreasonable resource utilization.
How do companies which do this keep the support workload of developers low enough so that they have time for development? It's a great idea, because as the linked interview with Wener Vogels proposes, it creates a developer that has more frequent contact with customers. I suppose eventually you have to shift resources around to start other projects, but then who supports what they're leaving? Even in large teams, eventually expertise will dwindle until you're adding new people to support an existing system.
I don't really agree with the original totem-pole idea that devs can do anything QAs and ops can do. While it may be true that a good dev is capable of muddling their way through any challenge because they know how to get around the numerous blocks thrown up by technology, there are definite skills to QA and Ops that developers may be lacking and which will lead to the dev being a poor substitute.
That said, I think devs have skills which are applicable anywhere in an organization. This goes back to the virtue of a programmer being laziness. Being able to see how to build systems and automate things is something that is potentially useful everywhere. In the cases of ops and product support, there is an especially large amount of value from having devs poking their head in from time to time to pick off easy targets for improved automation, reporting, logging, or what-have-you.
Yes. My company has tried for a good many years to keep product development partitioned from operations. We build stuff, through it over the wall, and let the ops team deal with it. Certainly, we give support where necessary, but it tends to be grudgingly given, and we never actually track time spend doing that support. I've been pushing, along with several other developers, that we at least need some sort of rotating product support team so we can work with ops to resolve problems in the field without disrupting our own timelines. We've moving in the right direction, but there's a lot of inertia to overcome.
> How do companies which do this keep the support workload
> of developers low enough so that they have time for
> development?
Of course, that's distinct from tier-1 customer support. "Devops" doesn't normally encompass client interaction.
It is in a company's (and the industry's) best interest to pair down well-paid developers and open up more entry-level positions, however, it's in the consumer's best interest for developers to maintain high accountability and "closeness" to the product.
I don't see these approaches as mutually exclusive I see how full stack development and silos can both exist in the same environment like if the Dentist in the counterpoint article's analogy takes over the hygienist's role when the hygienist is out on vacation.
It's another abstraction layer, and requires new skills.
You've hit the nail on the head there !
Ideally, people should support their own work. If I build something and have complete ownership of it, I'll make sure it runs. If I'm the one who's getting the money and career progress, and I'm picking the deadlines, I'll be the "3:00 am guy". It seems fair that it should be that way.
On the other hand, if I'm asked to build someone else's idea, to their deadlines, making technical compromises that I wouldn't choose, then I'm not going to be the "3:00 am guy" for shots someone else called. Fuck that shit, life's too short. If you want that from me, either pay a consulting fee ($250+ per hour) or give me serious equity (like, founder levels). Otherwise, That's Not My Job.
Salary jobs are for the stability of pay and work (i.e. not being the 3:00am Guy) and for career advancement (resume, respected titles, networking). That's what I'm paying for when I accept a salary 1/10 of what I'm worth to the business. If I'm expected to take on extra duties that don't advance my career, and not getting that stability, then I'm not getting what I paid for.
I'll gladly take the negatives of ownership if I get to partake in the positives (autonomy, self-direction, participation in the reward) but expecting me to take only the first is just unreasonable.