Managed services killed DevOps | Hacker News Reader

Managed services killed DevOps

techcrunch.com

92 pointsphillipchaffee148 comments

Software dev who just switched to devops recently. Good luck to whoever will manage an AWS infrastructure on his spare time between developing one feature and another. Seriously, the cloud is not shifting roles. Actually, the more everything gets moved to AWS, GCloud, etc.. and the more the level of abstraction will grow - the more you will need specialised devops IMHO.

Overall, disappointing article.

    > Good luck to whoever will manage an AWS infrastructure 
    > on his spare time between developing one feature and 
    > another.

The point of the article is that this work will be outsourced to companies that specialize in DevOps, that most companies that develop software will not need the specialist in-house.

(edit: Think about $MEGA_RETAIL_CORP. Their main-line of business is not software development, but retailing. Despite being a large company with deep pockets, they have difficulty recruiting the best and the brightest in the DevOps world to come and work with them. When $TURNKEY_DEVOPS_CORP approaches them and offers their devops as a service product, $META_RETAIL_CORP eagerly signs up.)

They're just moving the turtle lower. Now we'll have horror stories about devops outsourcing just like we have those about dev outsourcing :)

Aws/Gcloud bring DevOps and Developers closer together. We are able to work together with a better understanding of the tools being shared. Now I am able to set things up in the Development ENV, and DEVOPS can create a process to manage and automate the transition to Production. Sure I could write the scripts and tools, and it makes DEVOP roles more attainable, but it doesn't diminish the role by any means.

As someone who's worked on software, as a developer, and done operations for quite a some time, I imagine I could do both together quite easily.

Given that I've done this a while, I have a bunch of patterns and reusable things, that use to deliver something quite quickly. Once it's up and running, your job is to patch. Obviously developers like to be constantly running the new shiney thing, but if you put a foot down on that, stick to a low number of well known tools, and don't chase fads, you'd have a lot of time to work on features.

However, it also depends on where you are and what you're outsourcing. If you want me to start from nothing and decide we're using a whole bunch of tech I don't have patterns for already, it's going to be a while before you see me come up for air and do any features.

It also depends on what you're hosting though. Much of what we're selling is just a fancy CRUD app, they're easy to run.

It's Techcrunch.

And the easier it is to build up large systems that require an organizing principle to avert catastrophic failure from a developer that doesn't understand all the moving parts. (I say this as a developer. I know my limits when it comes to ops).

I'm one of those developers... it's manageable if you have the right skills and interest. I'd say I spend on average 2-4 hours a week maintaining our Deis PaaS on AWS. I'd be happy if we had someone working full time on this but we can't afford to.

same where i work. we have team of about ~10 backend guys and we share the load of doing 'devops'.

i would prefer having a guy or two dedicated to it, but we make it work.

Seriously. It's impossible to manage systems (ops) and develop at the same time if you have any sort of complex system. With AWS and such, you need permanent ops to deal with all the bugs, issues, and slow support. With a solution like Docker you need someone to set that up and manage the cluster. Either way, you need ops unless your infrastructure is very basic.

I think the reality is that it is somewhere in-between. Most companies dont want to spend on ops anyway, so they provision the least number of devops people and then complain that their deployments are slow or whatever results they are attempting are produced slowly (mostly due to emergent work due to ops struggles.)

I really like the goog model of your SREs spending 50/50 dev time and ops time, because it forces a shift in how you are deploying things and the ops people have a big say in whatever the devs are doing (not the case in many companies.)

It is not impossible to do any of these things, but you cant get around the basic complexity problem by saying "oh this is dead now, just pay someone else to do it." because the fact is you are giving over your keys to the kingdom.

For instance, People have been saying for years that "The DBA is dead." but I work with many companies that building new database driven products, thinking they can handle concurrency and growth, and then when the real complexity starts occurring because their abstraction is a bit leaky shit hits the fan.

An argument against specialization only works when you have completely replaced that industry with something entirely better (we dont have a lot of blacksmiths anymore), and that has definitely not happened for devops.

I disagree. The argument is about things like AWS Beanstalk. Once all of your deploys are going out with Beanstalk, most of the AWS management disappears. It's basically Puppet and Chef taken to the extreme if you use all of it's features. Once your one DevOps guy gets your scripts set up to easily deploy, what else is there?

Until you need to do one thing outside the Beanstalk pattern, then you're dealing with CloudFormation templates all the time.

No. You learn to use it just like you would a normal linux box - or you use their managed container service which is basically plug and play with docker compose.

Actually you don't or else you break the whole scaling / immutability model. I'm talking from production experience here. I ended up moving to OpsWorks and the level of abstraction worked better for me then.

I'm curious what you thought of OpsWorks? I tried it a couple years ago, when it was really new, and it was pretty painful. I'd rather just run Chef myself on regular EC2 instances and Autoscaling Groups. I recall a lot of issues hooking into the OpsWorks lifecycle at the right place, broken Berkshelf support, failures with no good error messages, an even slower build-test-fix cycle than usual, discrepancies between the OpsWorks cookbooks on Github vs what was actually run, and difficulty testing things out in Vagrant first. Has any of that gotten better?

This was a while ago too but I used AWS support and the Loft extensively to try to bang out what actually worked. Most of my stuff was pretty simple and I was a one person ops team so it was "good enough". I would not try to make it work with a larger team but still better than that generation of EB.

The commentary here is missing the point. The article is about managed services, not EC2, and yes, it is killing devops. My team just built a very successful ecommerce company doing respectable traffic volume with zero ops or devops.

We built it on Google App Engine. If you don't like GAE there are other PaaS providers that offer something similar. Developer life on a PaaS consists of building features day in and day out. It's not mucking with Docker, Chef, Ansible, Salt, Puppet, BOSH, apt, yum, or any other low-level systems management tools. This stuff doesn't even waste space in my brain[1]. Instead of learning this stuff, my team has been been learning about the business needs of our customers and building them out in code.

I plan never to hire a devops team. Running servers is like writing assembly language; appropriate only for companies with very specialized needs.

[1] Not quite true - I used to work on the BOSH team. It's a great tool for deploying CloudFoundry. But you shouldn't use BOSH, you should use CF, hosted by someone else.

I am reminded of a comment by someone famous:

"The first software companies had thousands of developers. The second wave of software companies had hundreds of developers. The next wave of software companies will have 1 developer."

Can't find the source, might not be the exact quote either, sorry.

I recently interviewed a PHP dev who asked me why we even have backend developers. From his perspective, we could just use Parse and it would do all the "magic" of the backend.

Then Parse shut down.

I think the lesson here is, you should never trust your core product to a third party. Also, when you use cookie-cutter services, you really limit the kind of things you can do, and vendor lock-in really, really limits your options in the long run.

I have heard this repeated many times and it's nonsense for two reasons:

First, it isn't actually true: Third-party managed services (say, exotic databases) are a mere network hop away. If you really had to you could call out to your own self-managed services in EC2 or whatnot. Even on Google App Engine, which is one of the most restrictive systems, we use a multitude of services running elsewhere in "the cloud". This is occasionally a mild annoyance, but it doesn't significantly alter the value equation.

Second, it's irrelevant: Migrating our application off of GAE would be tough, sure. You can call this "lock-in" if you want, but it ranks somewhere down a distant thousandth after a long list of concerns about my product and how well it is meeting the needs of my customers. Worrying about lock-in is a premature optimization.

The typical argument about the superiority of cloud services is that it's more hands-off, but what they don't talk about is how shared resources causes all kinds of new headaches I never had before.

I recently migrated from a dedicated DC to AWS because of business reasons (ie. CAPEX vs OPEX).

Before, my PostgreSQL DBs were always perfectly in sync because they were on a dedicated, private network. Now, I often get large replication delays because the network in EC2 isn't of the same quality I had before, so I've had to spend a lot of dev time working around the delay issue with creative caching solutions.

My point is, there's trade-offs and mostly my calculations show that they're not really worth it, if you have knowledgable Ops people. Which, once you grow to a certain size, you should.

I think you're misreading the message - managed services doesn't mean EC2. Managed services are things like DynamoDB, ElasticBeanstalk, SQS (in the AWS world) or Cloud Datastore and App Engine (in the Google world). If you're building on IaaS like EC2 or GCE, you will need ops people. If you build at a higher level, you don't.

If anything my argument holds up because you're putting even more abstraction between you and your systems by using the managed services, which leads to less understanding of how things actually work.

For instance, I went with running PostgreSQL instead of RDS because I have much more control over what's going on in the DB than I would with RDS.

But your point that the managed services don't require ops might be true in the short term, but what happens when something breaks? Or you experience a level of growth that expands past the capabilities of the services you've organized? There's hard practical limits to all the of SaaS offerings - mostly latency and throughput - that doesn't really come up in these discussions.

The analogy about it being like assembly programming might be more apt than I originally thought, given how not doing it yourself generally results in slower and poorly optimized solutions, but then "good enough" is often the mantra of the day at a certain scale.

Further on the "assembly programming" line of thought: there was a long time where compilers just couldn't match the speed of hand-coded assembly...but that time is almost completely behind us, and probably has been for quite some time.

As management tools mature and automatic, systemic optimizations are developed for SaaS systems, I wonder if the analogy will continue and areas where you need to manage it yourself to eke out performance and flexibility will get fewer and farther between over time.

It really limits your ability to react when things DO go wrong. Not having direct access to hosts limits your options a crap ton

It's ultimately a short-sighted choice - you get running really fast but then if you do well, you basically have to re-engineer from scratch.

It's basically the ultimate technical debt.

This is what I'm worried about, PaaS lets you get going very quickly but if you succeed...well I haven't heard great stories about scaling out PaaS.

The situation I'm in is our apps currently do low enough traffic to exist in a PaaS, but they weren't built that way from the ground up and live on bare metal currently. Migrating to say, AWS, would using ElasticBeanstalk be burdening myself with technical debt I don't currently have vs using something like CloudFormation (Terraform) and CodeDeploy?

Well, the question is - does that app have the potential to outgrow the SaaS offerings?

There's plenty of things that won't, for a long time, so you're fine to save the time since you won't need to re-do it later.

But if you can really see the product expanding to a large scale, you need to think a year or more into the future when planning your infrastructure.

I'm not sure that's always the case. If it's e.g. RDS, moving to your own Postgres cluster is doable when necessary

You hear this same sort of argument with the adoption of any new technology.

I imagine that when Flight Navigators started disappearing from flight crews in the 1970s with the advent of GPS and other related electronic navigation systems, there were those saying "I'd never leave the ground without a flight navigator - what if the GPS stops working?!?" Should we keep an old school Flight Navigator on standby "just in case"?

I remember talking to an retired mechanic about 20 years ago, who told me he never did a road trip without taking all his tools with him so he'd be ready to fix anything on the side of the road if need be. I wonder if he is still doing that....I doubt it.

As technology gets better and better, the "Number Needed to Harm"[1] needs to be considered.

[1]https://en.wikipedia.org/wiki/Number_needed_to_harm

edit: missing word

Do you have a SLA with Google? Do you have any plans to proceed if they just shut down GAE within of a few months?

This has happened with Parse, and this will happen again.

GAE is subject to Google Cloud's Deprecation Policy, which says you'll have at minimum one year to migrate off it: https://cloud.google.com/terms/

Which is okay if you just use Google’s Cloud as container host.

But if you have to rebuild your whole company, because everything you use runs on GAE, 1 year is almost guaranteed death for your company.

Google has been making significant visible investments in GAE, even with the advent of Compute Engine. There is also a hybridization of the two going on. While I can't claim to expect my code to live unaltered forever, I'm not worried about having an upgrade path. Also: I do have a lot of GAE apps that have been running nearly unaltered for several years. So there is a good track record.

For a point of reference, when Google rolled out the High Replication Datastore, they kept around the Master/Slave datastore for years and even built a pushbutton tool to automatically migrate apps. GAE has been around since 2008; this is not an experiment like Wave.

I absolutely agree. If you want to stay in the AWS world, look at Elastic Beanstalk or Lambda. I've been doing DevOps for years and most apps (internal or external) that I deploy anymore are just one Jenkins job that packages and deploys a Beanstalk version. Give Devs permission to Jenkins or have it auto-poll git and I never need to do anything else.

What about the team(s) that manage the underlying PaaS/SaaS Services?

Does DevOps or SRE's not exist in this realm?

DevOps is dead? Well I guess that means I can finally delete my fat repo of Ansible, Terraform, and Packer configs. Using AWS, GCE and others doesn't get rid of DevOps. You're just managing computers in someone else's data center instead of your own.

You definitely need to delete your Ansible, Terrafor, and Packer. Typical Linux distro is a lot larger than your project, but it works fine without that, at hundreds of platforms. Did you learn how to achieve such level of flexibility?

I don't understand the correlation you're trying to make. Linux is very sophisticated but it doesn't provision itself.

Linux distro is much more complex than application, but it installation is simple, because distro is split into lot of independent packages, which are wired using package manager. Lot of naїve developers are trying to solve same problem using imperative tools or develop their own package manager instead of learning of existing, well proven solution.

Oh you mean running your own apt/yum mirror? I've done that in the past. There's still configuration involved. You have to configure the packages that your machines then pull. You also have to configure the mirror itself.

For dependencies, use metapackages. For repository configuration, use "release" package with description of repository. Release package can be installed directly from http.

In my cases, I define few variables, then I will install repository definition and main package:

  export NODE=2
  dnf install -y http://..../staging-release.noarch.rpm
  dnf install -y node-backend

(and few other steps in some cases, e.g. putting of host names into /etc/hosts, or formatting and partioning of hard-drives).

It works fine on bare metal, vagrant, docker, aws, etc.

And how do you tell 100s of hosts to install that `node-backend` package?

It is called provisioning. How you created 100s of hosts in first place? Use same tool.

I'm pretty sure lots of developers are also using package managers, and sometimes private servers for those same package managers, but configuration management is still a separate thing that needs to be done and using tools to do it is very much a good idea.

I use same tools (package manager, services, scripts) for about decade. Tried to use Puppet and Ansible, but result is much more complex and it feels like I reinventing the wheel, because I need to implement things which I already have implemented in package manager, like validation, upgrade management, etc.

Difference between Ansible and package manager feels like difference between procedural programming and OOP.

Difference between Puppet and package manager feels like fight between two competing object oriented models.

Far be it from me to tell you where or how something should be implemented. If you're happy using packages then that's great! But the other two tools you mentioned using – "services, scripts" – are exactly what a lot of people find to be harder to manage than a configuration management tool.

Maybe your "services" and "scripts" are really sophisticated. If true, then perhaps you've implemented your own configuration management tool. If so, maybe you could share it and other people would find it useful too.

You are asking wrong questions.

The reason, why I don't use configuration tools is because I do no configuration in production. One of main rules for me is to perform all installation and configuration steps in controlled development environment, then test them in staging and only then deploy verified result to production.

Imagine that you are fully installed and configured everything at your new production host but in your development environment, then it powered off, deployed to production environment and powered on again and it works. It is result I want to achieve: deploy only ready to use software to production, in sealed form, so no installation, no tweaking, no anything. Like smartphone on contract. It can be achieved with docker, but docker is relatively new invention and my solution plays well with docker too.

So, no, my scripts and services are not a configuration management tool.

I think you are mixing the concerns of configuration managers and package managers. They are at different abstraction points. Step one: install software and its dependencies (package manager). Step two: Have all the software services in your cluster talk to each other (configuration manager). Step three is profit.

No, I am not mixing them. I just skipping step two, because it can be done automagically after step one.

It can. Is it efficient? No. This is so obvious that I'll have to attribute your opinion to lack of experience. Try deploying a stack with a dozen different services, designed with fault tolerance and horizontal scalability in mind, where each service is clustered, without a configuration manager. Then, review your opinion.

I have 25 years of experience. 19 years with Linux alone. Yes, I deployed stacks with dozen of different services. Typical distro has hundreds of services.

To anonymous coward, who downvoted me: if your large and "complex" application will be integrated into distro, so you will be able to apt/yum/dnf install it, will you still need your custom installation scripts?

`apt-get install nginx` only installs the package and some default configurations. You still need to manage the actual config files yourself.

What will you use? Bash? Python? Perl? Why not something like Puppet that keeps track of changes it needs to apply and has many useful features on top of it?

What happens if you write your installation scripts to use Ubuntu, but the order comes from high up to move to CentOS/RHEL? Did you write your scripts to be distro-agnostic? Because Puppet's `package` is.

Actual configuration can be installed using `apt-get install ngingx-site-config-prod`.

Now, try adapting that approach to do Blue/Green deployments [1]. Imagine how that is done using configuration managers. Can you see the advantage now?

[1] http://martinfowler.com/bliki/BlueGreenDeployment.html

I use it. Blue/Green deployments is one of main reasons to use package managers, because with package manager I can prove that two deployments are identical, including configuration files, which are also deployed using package manager. It was primary reason why we switched from configuration scripts to packages for provisioning and configuration.

Except that you'll need to reconfigure parts of the stack post-installation. You consider that an upgrade of the configuration "package"? You're reinventing the wheel. Doable, not efficient.

Why I need to reconfigure parts of the stack in first place? I am devops, so these things are automated.

In general, plan your changes ahead of time, test and automate them at staging servers, prove correctness of changes, and then deploy to production. Never do anything by hands (or by configuration tools) at production servers.

BTW: So nginx developers are using Puppet to install nginx? Because Puppet package is platform agnostic and we have nginx at every platform. (Joke).

I would hope that they have some sort of automation set up to test their code on multiple distros, yes.

You are misunderstands relationships between authors, packagers and users. Authors pays by code, packagers pays by maintenance, users pays by testing and with bug reports and patches. Testing of nginx at all distros and in all usage cases is not responsibility of nginx team. If you disagree, then demand your money back.

Even if you write RPM/DEB's to install all of your config files, you'll need something to trigger installation after you've updated your packages. How are you going to do that? SSH to every server? That doesn't scale very well. This is where other tools come into play.

One-time service and reboot of host or start service manually from kick-start script.

Reading this thread makes me feel like I'm looking at a conversation between sysadmins from now and one from the past. I'm not saying this to be dismissive though. It's possible there may be merit to what they're describing, maybe the emperor really isn't wearing any clothes. It just takes me back and feels weird to read.

If you are old enough, you should remember times when we need to enter network configuration manually, using network configuration TOOL: netmask, IP address, router IP, DNS servers, domain. Then DHCP SERVICE is invented: DHCP client asks DHCP server for that information and applies it automagically: it just works, for any device, now and in future, without human.

Ops tells me that they are using configuration TOOLs. I tells them that I am DevOps, so I use SERVICEs and SCRIPTs (clients) instead. They work automagically, without need for me to reconfigure servers manually.

Am I from the past?

Would you be opposed to publishing your architecture on github with minimal examples? The examples would need to be enough for someone to test out and verify for themselves. I think that would help people understand and contrast this against other solutions.

I agree.

I recently created a private cloud infrastructure on Digital Ocean. It would have been easier and faster (and more polished) if I'd gone with AWS, but then I would know how to use AWS, and would not know how to do these things myself.

Why learn about HAProxy, keepalived, tinc, etc when you've got AWS to handle it all for you?

Answer: Because relying on a service to abstract the inner workings for you is a path to the dark side.

Not sure why you're being downvoted. Abstraction is for ease of use, not ignorance as to what's occurring under the hood.

> Abstraction is for ease of use, not ignorance as to what's occurring under the hood.

I'm stealing this. I love it!

No. Abstraction is for higher level thinking. Dev Ops is still young so we are seeing many attempts at providing these abstractions but it is a very large domain to abstract over and thus hard to make a non leaky abstraction.

Given a non leaky abstraction, ignorance of whats under the hood is just fine.

What I was trying to say was more along the lines of a 3rd-party service abstracting the work is not a good idea in the long-term.

It is unthinkable right now, but what will happen with Amazon/AWS goes bankrupt and no longer exists? All these AWS Certified Engineers who are not actually devops will suddenly be lost.

On the flipside, if you learn everything from the ground up and then jump on AWS or whatever service is available, you will be in a much better position when the day comes to move away and to something else.

Abstractions in something like programming languages is fine because the language won't simply disappear one day - you'll evolve as a programmer and will be stronger for it.

I see what you are saying, and I agree with you that betting on any of the current abstractions is probably not a good idea.

But I really believe someone will figure this problem out, and at that point Dev Ops will be done. There is just to much money to be made.

> It is unthinkable right now, but what will happen with Amazon/AWS goes bankrupt and no longer exists? All these AWS Certified Engineers who are not actually devops will suddenly be lost.

Or they could just read this document and move to GCP.

https://cloud.google.com/docs/google-cloud-platform-for-aws-...

It's all virtualization/containerization/object|relational storage, no matter how you cut it.

Know the fundamentals, and you'll be fine. Everything else is rhinestone marketing.

Ignorance of what's under the hood is also called vendor lock-in, which is great for vendors but I can't see it ever being good for clients.

> Given a non leaky abstraction, ignorance of whats under the hood is just fine.

Minus vendor lock-in. Once you have your infrastructure dependent on ELB, S3, Redshift and all the other AWS niceties, you are effectively locked into Amazon. Locked, in the sense that migrating out has a relevant knowledge cost (deprecation of knowledge of AWS, need for gain in alternative knowledge).

It's not a dumb choice, but it's also not clear for either side. Running your own iron is much much cheaper. For my case (email service provider), our bill estimate on AWS would be about 300k€/year, while running our own servers costs 84k€/year (split at 48k€ depreciation and 36k€ operational costs).

> It's not a dumb choice, but it's also not clear for either side. Running your own iron is much much cheaper. For my case (email service provider), our bill estimate on AWS would be about 300k€/year, while running our own servers costs 84k€/year (split at 48k€ depreciation and 36k€ operational costs).

Something to note about US orgs: New depreciation rules allow you to depreciate $500K/year in hardware, in that year. That means that gear is yours, forever, whereas that $500K/year in opex to AWS is recurring forever.

can you link to a detailed analysis of that?

Did you want a link to the new depreciation rules? Or a specific scenario?

Link to the rules, and/or some analysis by a well-regarded accountant. I'd like to route the information internally. :-)

https://www.irs.gov/publications/p946/ch02.html is the relevant IRS publication.

Agreed. I wouldn't be on AWS if not for the fact that they're the only large, on-demand instance creation service that will also sign a BAA for our HIPAA needs. Even though we make use of the AWS specific tools like ELBs and VPCs, we still configure all the under-the-hood equivalents such as iptables, nginx, etc. ourselves.

WHY?!?! Just go to beanstalk and have it done for you or use Puppet/chef to manage it for everything.

I'm not sure if you interpreted that as configuring it all ourselves as in manually, but we use Ansible for configuration management.

It depends where you want to stop the abstraction, you don't write assembly code or your own database/web server, you are just drawing an arbitrary line on products as a service (and I'm not disagreeing with the sentiment).

I use cloud services because there's only one of me and I would rather be developing features. They also let me think at higher levels of abstraction.

I also like to think of using a cloud service as "hiring" the entire staff working on that product + benefiting from all of the accumulated knowledge that's been baked in. That makes a cloud service a huge bargain to me.

I'm interested to learn how you setup a vpc on digital ocean considering they don't even offer that.

They do, but it's very limited. Their private IPs only work within the same datacenter, so you lose the benefit of distributing your infrastructure globally.

The solution that worked well for me was using tinc to set up a VPN so all my servers, no matter what datacenter they're sitting in, can communicate with each other using private IP addresses (tinc has added benefit of encrypting, which DO's private network does not).

If you're needing help, I'm currently looking for more contract work :) Check my profile for contact information.

But you are still sitting on the same network as everyone else correct? That doesn't sound like a vpc.

This is a case of the wrong tool/service for the job. You need advanced networking and spent all this time duct taping it together on DO when you could have a real VPC on aws with security groups, subnetting etc. Not to mention locked in to DO's limited instance types.

Don't get me wrong, DO fills a niche for quick one-off type applications but it is absolutely no substitute for a real IAAS like aws, gcp, azure... Investing time in trying to turn it in to one is a waste and way worse than perceived "lock-in".

You also could have just as easily rolled your own setup the same way on AWS, so i'm not sure what argument is other than cost. (btw t2.nano instances are $4.75 a month) http://www.ec2instances.info/?cost=monthly

The key is to focus on the services that you can add value to over and above what the AWS managed services offer. There's no reason to futz around with building and maintaining a load-balancer infrastructure if Amazon's ELB does everything you need (unlikely, I will admit). AWS and other cloud services just provide an additional set of tools that may or may not be useful.

Plus also: $$$$$$$$$$$$$$$$$$$$

>DevOps is dead?

Fyi, the title is using the rhetoric template of:

  <SOMETHING> is dead.  Long live <SOMETHING>. [1]

...where <SOMETHING> is deliberately the same word in both sentences. The construct is a shortened version of: <SOMETHING> in its previous and more well-known form has evolved to <SOMETHING> with a new variation. That <SOMETHING> is still there the whole time -- which is sort of meta-joke connecting the 2 sentences.

It's as if someone wrote "Cars are dead. Long live cars." to signal the change from humans manually driving the cars to AI robots doing it. If you read the "cars" literally, it looks nonsensical because there are obviously cars existing both before and after the shift to automation.

[1] examples:

http://www.pcoitmurphy.com/_books_are_dead__long_live_books_...

https://www.tnooz.com/article/hotels-Airbnb-rented/

http://www.wired.com/insights/2014/08/moocs-are-dead-long-li...

https://en.m.wikipedia.org/wiki/The_king_is_dead,_long_live_...!

To add to this informative post: The original form of the phrase is "the king is dead, long live the king".

It was used when the king died in the sense of "the [old] king is dead, long live the [new] king".

So in that light it means "[the old way of] devops is dead, long live [the new way of] devops".

I would still disagree with that because DevOps really hasn't changed at all. It's not like AWS popped up yesterday. It's been around for a decade.

The article is muddled. It's like some came up with a title and then tried to support their conclusion with rambling thoughts without much cohesion.

Deadlines, I guess.

I think the analogy with QA deserves some credit here. "Testing" as a part of software process has not gone away, but the people who solely did QA are now gone - either evolved into doing both dev and qa; or moved to companies where the older processes still exist.

Testing was more closer to writing-code - so that got absorbed into the dev side faster. Disappearing QA was a you-blink-and-you-miss revolution.

DevOps is a little farther. Several developers are not able to wrap their heads around the deployment systems, the implications of hardware and network etc. But they will. Eventually. Because they are smart people.

I think the message of the article is that people who solely have DevOps expertise, should look to evolve to developing code as well. That way we have smaller, well-knit teams with each member having an additional speciality other than writing code.

My current job seems to disagree with you and their point about QA. As far as I've been able to see, the market explicitly for QA Automation is actually growing. The idea of

>Developers are responsible for full test automation and making sure their code works.

Is laughable outside of [insert new sexy startup here], if only due to the difference in mindset.

Granted I'm totally biased as a QA Automation, but the analogy actually removed quite a bit of credence that this article might otherwise have held with me.

Idk, QA Automation is growing but that has rapidly replaced TONS of "follow the procedure by hand" type QA specialists. People who knew how to write procedures well and document executing them well are disappearing in droves to 1-2 people who write QA Automation scripts in Selenium or something.

Yeah I can definitely see that side of things. The manual aspect of QA is definitely reducing, though I would hesitate to call it entirely dead. I love the fact that I've got experienced manuals who are able to craft well written procedures that I can automate.

There are more and more opportunities for QA Automation with the ability to write those procedures, which I suspect are the ones replacing the manual QA specialists you reference.

What I have noticed more and more often lately is people who say:

"Yeah, I'm in team X and I do the Devops part"

So we basically have an "ops" person sitting with the devs, and call the team a "devops" team. The devs have still no idea how to deploy their code. The ops have still no idea how the code looks like. But hey everyone feels better.

This is also true for many job offers in the form of "looking for a devops engineer". No, what they are really looking is for an ops person that has to sit in a dev team and do the part no one likes to do. s/He will be lucky if s/he can actually spend 10% of his time producing application code.

No, what they are really looking is for an ops person that has to sit in a dev team and do the part no one likes to do. s/He will be lucky if s/he can actually spend 10% of his time producing application code.

So? What's the problem with this? DevOps generally means someone who uses developer like approaches (e.g. scripting, automation, developing one-off solutions, etc) to solve operational issues. It is a role, and someone has to do it, and I would argue that there are people who love to do this role.

But if someone sought a job in DevOps thinking they'd be producing application code, they only fooled themselves. If someone sees some sort of status issue and forever tries to fight against their role, well that's just detrimental to everyone.

If you are solving operational issues, then you are doing ops (regardless of how much is infra-coding), imho.

I always thought of the Devops Engineer as a person who cares about their full stack, both application development and the operation of it, and understands how one affects the other. Something a step further than just infra-coding.

My title is DevOps Engineer and I still don't know what the hell that actually means – but who cares, I was hired to bring a new mindset, not a new toolset, to the development team I sit with. You're right, there's lots of devs here that don't know how to deploy code well or what configuration management is, but part of my job is spreading awareness and continually asking the question: how can we do this better. That is the spirit of DevOps, and that will never die.

What is much, much better than the alternative, where dev sits on their own room, with their own teams, and their own hierarchy, ops do the same, and there's a feud between both.

The title is a bit click-baity and misleading. DevOps isn't dead it's just becoming centralized.

The rub seems to be that quality managed services are widely available and affordable enough to obviate the need for in-house DevOps.

> The rub seems to be that quality managed services are widely available and affordable enough to obviate the need for in-house DevOps.

I have yet to find developers who are throwing themselves at carrying a pager.

DevOps isn't going away. Its just less relevant in smaller teams. Larger teams are always going to have an ops team.

DevOps does not need to carry a pager. DevOps will automate disaster recovery.

I'm a devops engineer, and I assure you, I have been on call for the last two years in my current position, attending to application and infrastructure alarms.

All disaster recovery cannot be automated. A sanity check is required before your Elasticsearch or Redis clusters decide to f___ their datastore.

EDIT: Or perhaps RDS is to your liking? We've had our replicas (off the same master) in AWS fail, at which point they would not further replicate. We needed to fail our app over to the master entirely, reduce its workload to not kill the master, terminate all of the replicas, relaunch all of the replicas, and then re-apply the workload back to the new replicas.

> All disaster recovery cannot be automated.

> The automated failover of our main production database could be described as the root cause of both of these downtime events. In each situation in which that occurred, if any member of our operations team had been asked if the failover should have been performed, the answer would have been a resounding no. There are many situations in which automated failover is an excellent strategy for ensuring the availability of a service. After careful consideration, we've determined that ensuring the availability of our primary production database is not one of these situations. To this end, we've made changes to our Pacemaker configuration to ensure failover of the 'active' database role will only occur when initiated by a member of our operations team.

Pinterest as well uses manual intervention:

https://engineering.pinterest.com/blog/open-sourcing-pintere...

All disaster recovery in general case cannot be automated, you are right. But it definitely can be automated to the point, when problem can wait till next Monday.

To recover any disaster automagically, you need five things: your actual data, your working software, a working hardware, a working network connection, money. All that, except actual data, can be prepared ahead of time.

If your datastore can run in a degraded state through a weekend, its not a critical datastore (or you're shedding writes or making other consistency compromises).

If you have a single point of failure, then it is your fault.

>>DevOps isn't dead it's just becoming centralized.

Should be fun for the Fortune 500's. Part of the justification of moving to the cloud was laying off the sysadmins and other people that did "DevOps" before it was called that.

Once it's centralized, the human instinct to "ensure survival of the group" means the DevOps team will get bloated in headcount, budget, tools, etc.

I assume that will be followed by some new wave of features called the "AI Cloud" or something that manages itself, allowing you to now lay off the new centralized DevOps team :)

Google SRE and Netflix Devops are places where Devops is not a mindset or culture but type of engineer that requires operation and development skills.

The reality is the field is less welcoming of specialized developers. We all have to develop, test, and deploy our code. Call yourself whatever you want but make sure you comfortable doing it all.

I've started reading the Google SRE book that was just released, and now I know more about the people I have to work with when the situation arises. Liking it so far :)

http://www.amazon.com/Site-Reliability-Engineering-Productio...

I love that the screenshot is HTML. DevOps is known to be very heavily HTML based work... .

In a proportional font. Ew.

One funny things I noticed about DevOps culture is that it is usually presented as developers should do more ops. I never quite saw it go the other way, encourage existing ops to do more development. Wonder why that is.

Same thing for testing, same thing for QA. The reality is there's much less room for specialties.

Once upon a time there were programmers.

Then there were systems programmers and application programmers. Systems programmers wrote operating systems and utilities for them. App programmers wrote apps. There was a lot of crossover.

Then there were operators, systems programmers and application programmers. Operator was a junior position who did physical things (mount tapes, plug in cables) and ran commands to do things on the systems. They usually moved up to being...

Systems administrators, who did some programming in service to the systems, but not not much. The more senior a sysadmin was, the more time they spent programming and the less time they spent doing physical things... unless they wanted to do that.

Sysadmins started to specialize. People who configured switches and routers and talked to telephone companies became "network engineers". People who spent time working on firewalls and security policies and thinking about that became "security engineers". Junior people who read scripts to end users became the helpdesk. And so forth.

Then we noticed that a bunch of people were doing things manually when they should be automated. This was especially bad in places where there were no senior sysadmins or systems programmers. But we did have the internet, and senior sysadmins got together and started writing tools to make their lives easier: infrastructure automation.

You probably know the story from there, but I'll wrap up with one more important point: you know how when writing a business application you need to have a subject matter expert who actually knows what they are doing? Operations is exactly the same way. All the automation in the world won't help you if you don't have someone around who knows what they are doing. Some people can outsource this to "the cloud", but not everyone.

That's because if you tell a sysadmin he needs to do development he says "yay!". Not much encouragement needed.

That hasn't been my experience from the folks that actually grasp the concepts. As an ops person myself, I spend the bulk of my time writing code. It just isn't product code. It is code for an internal customer, the devs. Sometimes I do small commits to the product codebase, but not often.

I think you see devs make this comment cause they think they are doing "ops work" instead seeing it as consuming the ops team's work. It isn't about everyone doing equal code commits. It is about speeding up things across the whole business. Every ops person you hire is money you don't have for a dev. It is more efficient to enable the devs than it is to hire ops people to do a small portion for their job.

Same goes for QA. The groups that have really got the message are including QA teams that handle much of the testing via automation. A good QA team writes code. There is plenty of development going around. It just might not be on product code itself.

Devops and QA are more alive than ever. The article actually makes that point. It is just that the boundaries between the disciplines are being blurred consequent to the erosion of distinctions between hardware and software, testing and deployment.

It would be more precise to say that the era of the code illiterate sysadmin or QA engineer is coming to an end. So is the era of the developer who didn't understand the basics of operating systems and networking.

This is based on the premise that DevOps is a team of people, it's own silo. And that is not what DevOps is about IMHO...

yeah, if your company has a devops team you are really just doing traditional ops / devs, but with a split ops team

I was surprised to see that most of the comments here seem to take DevOps to mean simply Ops. The whole point, the only point, of devops is to not have "write feature, throw it to ops"-processes. I think the latest generation of PaaS can been seen two ways: on one hand, it's like having a great ops-teamt that will handle everything you throw at them (and have set up clear borders for what is "their" responsibility), the other is to say that it's great for devops: with the low level of management needed for cloud infrastructure (compared with setting up your own hardware, choosing and tuning a network storage solution, manage fail-over for databases, making sure to patch ssh/TLS/libc/libz/libtiff/libpng...) means that we get better DevOps: you might prototype in a local docker, but as part of the commit cycle, every dev will run tests, and push out to production, with rolling updates.

I don't know when "DevOps" became so watered down -- I guess it's the same old story as with Lean/Agile: the real win is in changing the business process, and has nothing to do with tooling and consultants: changing processes is actually hard, so no-one does it. But when you've paid out the nose for "educating" your "team" to do agile/devops, you start to claim that your waterfall was built by Toyata. Because you paid for the privilege.

To me, containers seem to be a perfect way to have a reasonable contract between Dev and Ops. The Ops team just has to maintain a secure, stable, and performant infrastructure that can run containers, and Devs can put whatever they want in the container, and not have to worry about anything below it.

Sure, but now dev has the responsibility for upgrading all those libraries again. Just because your container worked fine 5 years ago, doesn't mean it's a good idea to run the same container in production today. Maybe it parses png images, or handles ssl termination.

Or you could say that ops now has the responsibility to scan all containers for old binary code/bad behaviour, and throw the container back over the fence if they find an issue.

I do agree that containers are a good start on having such a "contract" -- I'm not sure it's enough.

DevOps is the anti-silo... It's funny and a bit ironic that traditional corporations have heard some vague information about DevOps and immediately decided that they should create a DevOps silo/team, while apparently missing the fact that the entire DevOps philosophy is about breaking down silos and having everyone build Ops into their codebase...

QA is dead because Agile killed it? (quoth the article) I work at a Fortune 10 company and this is just wrong.

Furthermore, leveraging QA in an Agile process is critical to continuously improving our software quality.

Exactly. QA's role has definitely shifted (or should shift) toward being involved earlier in the process and helping to define requirements, but it is by no means dead.

Ops teams are more useful when they're analyzing data and improving architecture based on the results of their analyses. The ideal ops role is beginning to phase out the "overqualified support engineer" framework and usher in the new idea of being an "infrastructure architect".

To me, this article is more about the architectural/business decision of outsourcing as much as possible to 3rd-party services instead of reinventing the wheel in-house. How many Chef repos have the same stupid passenger cookbook vendored in? How many times have you had to fuck with that cookbook because it broke or something got updated? My websites for http://psychedeli.ca, http://brother.ly, and http://www.waxpoeticrecords.com are hosted on Heroku and, apart from the occasional Redis issue, they've been chugging along quite smoothly. I've had little to no issues with my development and continuous deployment process, which give my apps the same safeguards that I get with work (even more) with no one needed to "watch over" the project...other than me of course.

DevOps has never been more popular in the whoishiring thread. I run monthly trends on the postings, and DevOps just entered the top 10 of terms used[1].

[1] http://www.ryan-williams.net/hacker-news-hiring-trends/2016/...

"why not simply teach all developers how to utilize the infrastructure tools in the cloud?"

You need someone to liaise with the Cloud company, set everything up, test it, organize changes, and update the system regularly with new functionality. Even if a Dev had 10 years experience managing infrastructure, they still shouldn't be doing it because they should be developing products. If only there were someone whose job it was to manage Operational changes, and liaise with the Developers... Hmm.

And by the way, not everything is moving to the Cloud. We've had managed services in one form or another since the early 90's, and the result is always the same: the service provider is blamed for things working shittily and the company brings services back in-house.

And his continual refrain that "QA is dead" - does this guy understand product development lifecycle? Whether you're designing a brand new product or just bringing out a new version, you have to have QA. Can you imagine a medical device or medical web service being developed without QA, or QC?

Thanks for the troll, TechCrunch.

Devops is dying. And the secret weapon is not mentioned in the article (Docker). With Docker and a scheduler (swarm, kubernetes, etc), what are you going to use Chef or Ansible for? Just to install Docker on a Linux instance? Docker is just getting started, and we can already see that they are going to make a lot of tools and companies obsolete.

Even with Docker, I don't think that you would prefer to configure the environment yourself rather than use dedicated tools like Chef and Ansible, assuming your environment needs are complex enough.

So, DevOps as you (mis)understood it is dead. Long live DevOps as it was meant to be understood.

Look, I appreciate what people are trying to say about this stuff, but there's one universal truth that managed services absolutely cannot get around: Cost. As someone that's bootstrapped a couple of businesses from zero, I have had to lean heavily on my ability to work with bare metal. It is an incredibly valuable set of skills that translates literally to more money. Managed services will not kill off DevOps because there will always always always be companies that want to be profitable sooner, or don't have a ton of money to throw at AWS.

Second, this is a really clickbaity article.

DevOps != Ops.

You don't have a DevOps guy, or a DevOps team. You have a team that does both development and operations. The less work they have to do (like the reduction of effort with managed services) the better, but it's still work we need to do.

If you call out DevOps as a specific role, they you're not doing DevOps. Just like those practicing "Agile" are likely lacking in actual agility.

As someone with the label "Application Developer" who's been having to develop and deploy and learn my company's entire backend without much help other than "oh, here's the username/password for that, here ya go" from my superiors in IT, I don't know what to think of this.

The servers and databases are all managed by my one boss, but I'm basically now responsible for every aspect of my application's lifecycle. While I'm a 1-man team now, I know down the line we're looking to expand and I'm trying to set things up right for when that day comes - proper version control, and an automated way to push updates to my Django app.

I was looking at Docker and the likes, but we're unfortunately running Microsoft SQL Servers and Windows Servers which apparently aren't going to see much support for virtualization/containers til the SQL Server 2016 version comes out. My company is an engineering firm, not tech, so paying for an AWS bucket or me making a case to get the fuck away from Microsoft backend stuff (the headaches I've had with MS SQL already...), etc etc probably isn't high on their to-do list.

Maybe due to the sort of company I work for this article just doesn't apply to me anyway, but it seems like the author assumes any company rolling out software or a service is doing so at the cutting edge and as its primary focus.

I'm not sure my struggle to find a sane way to automate our deployment and updates would be helped too much by literally any of the cloud based services anyway. Even if they gave me a green light to go all-out on whatever service I wanted to use, at the end of the day I'd also have to manage all of those services and their connections to the app as well. I'm trading off some work for another kind, and a lot of these cloud solutions are hardly painless.

If you automate sysadmin then you can make the case that sysadmin is dead, not the people who write the automation.

Just one thing taken from the article comments:

> HTML used as the screenshot is as horrible as article.

looks around my devops office

I guess our company didn't get the memo?

In my experience developers have had to set up continuous deployment environments. If you design a set of cloud resources for your application in a dev environment, you are the best person to script that deployment into another environment. So the developer of the application ends up having to script the deployment as well. DevOps always seemed like too much communication between team members for too little benefit.

Reply on news.ycombinator.com