Is there trouble brewing in the land of DevOps? | Hacker News Reader

Is there trouble brewing in the land of DevOps?

blog.gardeviance.org

30 pointsEwanToo19 comments

Nothing new here really. Shitty architecture and practices are shitty architecture and practices. If you're not disciplined enough about how you package, deploy, and manage the lifecycle of your software then putting the blame on technology X or methodology Y will not change anything. The root cause of the problem is never the technology really. The root cause is almost always the human processes that lead to creating poorly documented and buggy software and sprinkling magic DevOps, agile, or whatever dust over things doesn't make a difference no matter how much advocates would like to believe otherwise.

A concrete example is that even before containers I saw plenty of binaries that were magical black boxes that no one knew how to re-create from source anymore, well not without some human intervention anyway by poking and prodding things at certain points during the compilation process. So to say containers and microservices are making things worse or better misses the mark. Things are the same as they've ever been and the more disciplined software shops are still doing just fine. The less disciplined software shops are still finding plenty of ways to blow off their feet (referencing a quote about C++). DevOps, agile, lean, six sigma, or any other methodology has nothing to do with it.

A phenomenon I've noticed is some believing that containers solves configuration management. It's true that containers through being immutable means you don't have to worry about what cruft has built up from previous changes (which is a big win). That still leaves you with the core problem of how you manage your configs.

The core of configuration management is a) how you express the state of the world and b) how you move from state to state safely, with potentially hundreds of changes in flight at once. For the former I'm excited about projects like https://github.com/apmasell/flabbergast, and the latter is mostly a matter of breaking rollouts into individually rollbackable changes combined with good policy.

More specifically I've noticed less reliance on configuration management frameworks like puppet and chef.

If you're building a container cloud, consisting of 'immutable' servers and a scheduler like kubernetes, then there's not much of a role left for one. You might use the lightweight versions (e.g. chef-solo) to bootstrap the servers, bake an AMI (or god forbid a container image). And you'll use something like cloudformation or terraform to configure the cloud. But if I'm going to toe the immutable infrastructure line I'm not going to use a config mgmt tool to update the state of my existing servers.

I've only just glanced at flabbergast but in terms of readability at least - it seems far short of something like Ansible

https://github.com/apmasell/flabbergast/blob/master/flabberg...

(I'm not arguing that they solve the same problem - I really have no idea - but readability matters and choosing a JSON syntax over YAML or similar when humans are meant to be parsing it seems like a mistake)

Flabbergast is solving a different but related problem, it's about expressing the state of your system. Lots of times you want to say the dev environment is like the prod environment, except for these 3 things. And by the way, anything in the European region needs different settings too that interact with dev/prod.

I'm a fan of Ansible, but it isn't really suitable for expressing multi-dimensional and multiple-levels of exceptions, from experience it works for about 2.

How that state is actually rolled out is the job of something else (which could be Ansible or the update portion of your cluster scheduler).

Fair enough but does that affect my thoughts on readability? YAML is actually a superset of the capabilities of JSON so as far as I know there's no inherent advantage to choosing JSON as the syntax.

Thing is containers themselves are not immutable. One implementation (Docker) enforces this but it's not so when using raw namespaces/cgroups, systemd-nspawn or lxc/lxd etc.

IMO it's not tools that are the problem but a lack of deep understanding. A lot of people practicing "DevOps" lack the raw knowledge.

This I can agree with. It almost seems as though this new kind of "DevOps" (to use the same air-quoted term as you) is being used as a band-aid for companies to be able to accomplish traditional sysadmin goals, without having to pay for people with depth of knowledge. Things like Docker and cloud services make it easy to think that one does not need the knowledge a sysadmin needed to have ten years ago, but that's not true. What's happened is that applications like Docker have obscured the need for that knowledge by making things easier to accomplish. And then when those companies run into problems, nobody on the tech team knows how to put out the fire. Now, I suppose that's not such a bad thing if you're outsourcing all that work to a cloud services provider. Then it's a matter of opening a support ticket. But then again, you're at the mercy of those providers. I can see why Docker is so attractive for companies, which implies one can self-host these cloud services and save a ton of money, while also appearing to not need that raw knowledge because of ease-of-use provided by applications like Docker. It looks like an attractive way to cut down on quite a lot of cost. But what happens is that the cost is actually offloaded from needing the knowledge to build and maintain the infrastructure, and onto the moment of failure. And the thing about that cost offloading is that this cost could kill your business. Just picture a tech team with red status pages and buzzing phones, but without the knowledge to dissect what's happening. It's a ticking time bomb. Also I should mention I'm saying this as a huge, huge fan and user of Docker, but also as someone with twenty years in the industry, watching the "new DevOps" crowd.

You're 100% right about discipline. I work for a very undisciplined and border-line incompetent company that decided to roll out a microservices architecture, continuous delivery and devops culture into a behemoth enterprise product. This has simply been a disaster because the incumbent methodology has been hack it until it mostly works.

However, with respect to technology, some technologies do not lend themselves to these processes. .Net I'm looking at you here; you're a tangled build nightmare, a deployment clusterfuck, require too many resources to lift light tasks and most of your core as API is a mess of obsolete ideas,impossible to test abstractions and a bunch of dubious hand grenades (ReaderWriterLockSlim for example which has deadlocked on me numerous times and is the basis for concurrent collections).

We do just fine w/ DevOps on .NET. We auto build on push, publish a versioned Octopus Deploy package to our TC NuGet feed,and deploy to our dev environment many times every day. We haven't had to actually touch our build/deployment process in over 4 months, except to click the single button to move from environment to environment. It really is fantastic.

That works for simple deployments but when you start having to deal with signing, licensed components and stuff like ours (55 deployable packages) then it gets hairy. Also managing windows infrastructure is hell.

Fair, though we've got a few things now. We're all on azure and all our stuff is @ platform level, and built to throw away, so it's a bit different.

Ok you're lucky to be on Azure PaaS!

I agree with you, but containerization gave us the ability to counter 15 years of technical debt.

I talk a little about this here:

https://www.youtube.com/watch?v=zVUPmmUU3yY

Doing it, though, requires attention to exactly those things you outline. But the technology can make it a more tractable problem, solve-able in this case by three people working in their spare time.

ShutIt is great. The reason I say that is because I've already built 3 of these things with a similar philosophy, which is to say that bash scripts are perfectly adequate when coupled with some high-level orchestration/lifecycle mechanism that is managed with Ruby or Python.

I'm definitely going to give ShutIt a try because my next gig is a Python shop and I'm not really interested in writing another custom tool.

You should definitely evangelize this tool more because every single shop builds something like this but it never ends up being open sourced. That's a lot of wasted effort across the board.

Yep. The more i read about containers and devops, the more it feels like its about making sysadmins subservient to programmer whims rather than custodians of server stability and availability. Or perhaps a "webification" of software development in general.

Or perhaps one can liken it to the old story about the turtle and the hare. With programmers as the hare...

There's always been a gap between the devs and ops. And one of the core side effects of this gap is the knowledge transfer problem. With the devpops movement you have devs doing production deploys where the ops (or even other devs) might not know exactly what's running out there:

"some IT landscapes contain components that people don't know how to recreate especially since the person that created the component has left the company" , "if we consider multiple layers of microservices then it can be quite disconcerting that some of those lower order component layers maybe in a position that we cannot recreate".

The knowledge transfer problem between the devs and ops is not really a technical problem. It's a human communication problem. With the new tools (like containers and microservices) it manifests itself in a slightly different way allowing you to deal with the communication problems further down the line (because you can deploy a black box container without breaking other things running on the same machine/VM), but you still need to transfer that knowledge.

I'm curious if there's a market for tools and products that automate this knowledge transfer...

This article was quite hard to read beyond the pretty wanky diagram:

Culture: Inertia vs Fluid / Gameable Corporate Focus: Profit / Disruption

I'm sorry, someone has been reading far too many startup articles, it's like a parody.

Buried deep within the self-fullfilling analysis, corporate buzz words, and stale metaphors is a simple concept: the early adopters of containers implicitly embrace NoOps, and eventually the absence of explicit dependencies and documentation will drag organizations down.

Reply on news.ycombinator.com