Three Ways of DevOps
ermetic.com
ermetic.com
By the way, I would recommend not trying to make your systems clever and “learn”, as the Third Way™️ suggests. Keep them dumb and understandable. The last thing a critical service needs is mysterious, unpredictable, unexplainable behavior. This point alone is a huge red flag for me, brace yourselves for five years of over-engineering as people blindly latch on to advice like this.
I suppose this is the next round of silver bullets after “microservices”.
This article doesn't really explain The Three Ways that well. I recommend going to the source: https://itrevolution.com/the-three-ways-principles-underpinn... Keep in mind this is only one of several ways of explaining what DevOps is about, as the whole concept evolved over several years.
Microservices were old and busted 5 years ago, but like Scrum, the industry pays no notice to broken patterns, only new ones.
"Learning" is definitely a higher-order pattern that you have to practice to get good at. Whether it's machine learning, distributed tracing, sliding metric alarms, etc they aren't easy to implement correctly but they can level up the efficiency of dealing with big systems. You're right that simpler systems are best, until a specialized team can work on more complex ones without getting run into the ground.
First Way - is whatever practice, I never can remember. Second Way - is another practice, no idea which one. Third Way - is the third practice, naturally, equally eludes me.
I feel you. Agile / DevOps certified consultants are an oxymoron. Outside of evangelization and some advice, they have no clue of the project and as such are just outsourced bosses. The core concepts are not that hard to grasp: teams are responsible to deliver value in small batches to the client so they can get quick feedback from the client. Having to employ them should be seen as a failure and as a symptom of a dysfunctional organization where the teams are not trusted by the higher ranks to have a say in how they organize themselves and thus they hire expansive people who gave them a sale pitch that corresponds initial vision. tl;dr: agile comes from within, people telling you how to work are just other way to say "boss".
> By the way, I would recommend not trying to make your systems clever and “learn”, as the Third Way™️ suggests. Keep them dumb and understandable. The last thing a critical service needs is mysterious, unpredictable, unexplainable behavior. This point alone is a huge red flag for me, brace yourselves for five years of over-engineering as people blindly latch on to advice like this.
If that happens the problem is not the people who try to develop improvements, it's the system that is not resilient to changes. These "mysterious, unpredictable, unexplainable" behaviors rarely manifest themselves all at once just by magic. They are usually caused by a comorbidity of problems that has accumulated due to a lack of will to fix them promptly or the lack of tools to catch and identify them early.
Blaming your developers because they attempted to improve the system is just downright counterproductive and demoralizing because you will end up with a Tech Lead™️ playing police because they think they must protect the System like the Secret Services must protect VIPs from threats that could come up in all directions.
Instead ask yourself those questions: How many times have you heard "we don't have time to fix this"? How many sheets of paper would you use to explain the multitasking architecture of your system? Can you test your system on a single thread so you can make sure your sequential logic is sound? How much time does it take to run all the tests? How much does your development environment differs from production? Can you easily toggle a feature to test it?
The problem is not the "3rd way". There is a reason why they have numbered it from 1 to 3, it's because you can't achieve the third without the second and the first before that. In other words you can't learn acrobatics without learning how to walk first.
> I suppose this is the next round of silver bullets after “microservices”.
Fads are universally bad yes, but as with most prejudices they always have a kernel of truth.
I was replying to the descriptions in the article:
> The third way of DevOps is continuous learning. [...] Sometimes this means introducing artificial intelligence and machine learning into your systems or tooling, to ensure that it is continuously learning.
> The third way involves continuous learning. If our automated monitoring system, itself, can learn continuously from the traffic, access, and behavior of our cloud activity and users, then that respects the third way. A system that can train itself, learn when to remove permissions, when to replace permissions, and when to block access, not only respects the third way, it makes for a fantastic least privilege security tool.
If your company specializes in this, and you have ML experts who know how to design, maintain, interpret, and monitor the models you are using in production, and who carry pagers so you can page them at 2 AM when something breaks and you can't understand what the model is doing...then sure. If you are the 99.999% of other companies, then this is a terrible idea and you should let me know before the interview so I can avoid working at your company (unless you're hiring me to remove it, because it inevitably caused more problems than it solved).
Excited to read this article and after briefly skimming, didn’t see any immediate reference so figured I’d leave webcrumbs here.
The Goal: https://a.co/dUDdGQ6 Phoenix Project: https://www.google.com/search?q=phonix+prijext&ie=UTF-8&oe=U...
The Goal actually specifies which misconceptions we have in many cases that leads to solutions that seem reasonable but are completely wrong! Without spoiling the whole book, but do look for the explanation in there for "statistical fluctuations" and "dependant events".
And then the Goal make you the reader come up with a solution that fits the factory environment (with the stat.fluctuations and dependant events) like a glove. A solution that creates a sort of a miracle for that time and place.
Other Goldratt books give more example, and more teachings, on how to look at practices everyone is doing wrong and appraising those using common sense. A favorite saying by Goldratt is that "common sense is not very common".
So just "solving problems" in an IT environment in most cases will lead to the wrong solutions, just because many people assume that solution X is the best solution without even looking at the problem THEY have. (GitFlow I am looking at you!)
Goldratt's books are pure gems, most of them I have read several times over the years just to remind myself each time that I have not understood something properly the last time I read it.
Automatic changes are ticking time-bombs which rarely have any place in production. And this particular example would break production very quickly. For example, a system that performs daily and monthly reports will break when the automated system removes the permissions it uses only once a month.
It doesn’t follow from wanting to speed things up for other teams that you can avoid some kinds of testing in the pipeline by having monitoring. That makes no sense.
It also says a full test suite in the pipeline is not needed because there should be monitoring in production.
Monitoring doesn’t catch all bugs, and letting things fail at a later stage isn’t the right path for a CMM level 5 application that lives and/or serious investment depend on.
Granted, while application monitoring puts additional ongoing load on the application, which slows it down, going against the desired goal of speed, it can help to have application monitoring that may identify slow, insecure, or error prone queries. However, that shouldn’t have to run constantly unless it has low overhead. Also, if problems are just reported after deployment and aren’t a gate to deployment, then you’ll keep pushing crap code into production.
So, what I’m reading here is that Dev Ops, as defined, values getting things through the pipeline quickly, so that when bugs, security, and performance problems are found, those aren’t a Dev Ops problem, and those production issues can be handled by developers, because they’ll have fast feedback from monitoring that maybe they didn’t kill someone yet?
o if you can use something premade, use it
o if you can buy in the service, buy it in (with in reason, make your own buy/build checklist)
o if you have to automate it, do it, then opensource it
o If you have made it, document it, so you don't have to answer questions.
Dont' communicate one-to-one, but always on an (internal) public medium, preferrably more persistent than chat.
- Need to explain something to someone? Write it a wiki and send them the link.
- Have a great idea that needs brainstorming? Document it as a proposal and open a pullrequest to discus and have it merged into architecture documentation.
- People asking questions about your code in a pullrequest? Don't explain in the PR why you did it, make your code and comments speak for itself so questions are not raised in the first place.