The Fallacy of Move Fast and Break Things
launchdarkly.com
launchdarkly.com
- "A ship in harbor is safe but that's not what ships are made for."
- "If I've made more shots than you, it's because I've _missed_ more shots than you." -- variations of this from Michael Jordan, Wayne Gretzky, other athletes
- "All great writers got better by writing a lot of _bad_ sentences."
- "If you want to learn how to ride a bike, roller skate, ski, snowboard, etc you're going to have to _fall down_ a lot."
All of them are trying to be provocative in the same way: expect repeated failures when experimenting/exploring what success looks like.
So MFABT when compared to variations of previous quotes is pretty standard advice. Nevertheless, everybody likes to poke holes in it because it came from a source a lot of people don't like: Mark Zuckerberg & Facebook.
Should programmers[1] at NASA "move fast & break things"?!? Of course not. I know what the original context of MFABT was about so it doesn't apply to NASA.
[1] https://www.fastcompany.com/28121/they-write-right-stuff
Most of the criticism of "move fast and break things" is word thinking. The move fast part includes "rate of correction".
It is not fair to characterize all “Move fast and break things” as falling into this egocentric bucket, but My anecdotal experience is that this is what attracts the most criticism.
> Ever tried. Ever failed. No matter. Try again. Fail again. Fail better.
There's also context to it that matters -- everything from the great documentation diaspora from a decade back where they blew away a developer wiki that represented years of community effort to describe how to use the various APIs without having anything remotely like an adequate replacement, to changes a few months ago which temporarily had access to the "Friends Lists" features unavailable which made managing privacy settings more difficult. There's a lot of things FB does right too, but when looking at blunders, it's easy to have the catchy phrase come to mind and wonder if maaaybe someone overconfidently strode forward MFABT in mind when more caution might have been warranted.
You can't make an omelete without breaking a few eggs, as they say, but if you say that often enough, you start to get some people who think cracking a few open means breakfast is ready.
No, it only looks like a desirable "imperative" if you read it with a hyper-literal interpretation.
MZ's aphorism is just using the rhetorical device of taking something undesirable and acknowledging it. Yes, the literal words might appear like an "imperative command" but the underlying meaning of the message is actually "failures are the path to success". The rhetorical language is a form of reverse-pyschology.
Another similar example is salespeople or romantic suitors trying to get "yes" instead of "no" from prospects. The "no" is undesirable but rejections are inevitable in the process. Thus, one reverse-psychology approach is to frame it as, "How many _rejections_ did you get today? Only 10? Go make some more phone calls until you get 50 more rejections!"
It still doesn't mean rejections are the real goal. We're just humans playing with language to get past the fear of rejections holding us back.
"Break things" is objectively an imperative. There's no hyper-literal or otherwise careful parsing necessary to arrive at that realization. It's the default.
Careful thinking is actually what you need to arrive at the more useful non-literal understanding. And unfortunately the phrase itself doesn't encourage that.
Something like "Move fast, even if there's risks" or "Prefer momentum over risk aversion" would be the face-value formulation.
> We're just humans playing with language
And being played with by language.
If language is powerful enough to orient someone on the benefits of preferring momentum over risk aversion, it's powerful enough to distract from the threshold where there's inadequate anticipation of consequences.
Any given formulation may do one job better than another.
MFABT is not an optimal expression. There are better ones.
I wrote desirable imperative.
You are focusing on grammar (objective interpretation). I was talking about the meaning ("imperative" as synonym for "importance"[1] instead of grammar).
Likewise, "go get more rejections" is also an imperative (via objective lens of pedantic grammar categorization) ... but that's not what the underlying message means.
>MFABT is not an optimal expression. There are better ones.
It may have been fine as an internal mantra within the walls of Facebook when it was a smaller private company. The engineers would know what it really meant. But then MZ publicized it in an investor letter before the IPO which opens itself to misinterpretation by the outside world like journalists, bloggers, etc.
If a fishing ship goes out to sea and fails to catch enough fish or is wrecked because of a storm, it affects the families of the crew and everyone else who depended on them.
If Michael Jordan fails and misses the game-winning shot, it affects his whole team. If it's losing a playoff elimination game, it affects the workers at the sports arena because they don't the get the income from extra games.
> With MFABT the party doing the breaking isn’t often the one suffering from the breakage
I don't know about every company that's (misusing) MFABT but with Facebook, it was their mantra before 2012 when they were not making any profits and trying to survive while competing against the bigger website, Myspace. Their software engineers could have conceivably broken their website so badly that FB goes out of business. That's what happened with Knight Capital when they misconfigured a server and lost $450 million.
MZ's aphorism is just a twist on a common observation about making progress. He expanded on it in an interview:
>We've optimized so much of our culture around just making it so that people can come and build things quickly. Whether it's everything from having the right tools in the right development environment to build things quickly, to nightly code pushes, hiring the best people who have a bias toward just pushing things very quickly, very entrepreneurial. The whole culture is tuned around that. And I think there's probably something in that for other entrepreneurs to learn which is that making mistakes is okay. At the end of the day, the goal of building something is to build something, not to not make mistakes. [1]
He actually does not want a broken Facebook system. But he also knows that if there are no mistakes, it means the team is not bold enough with adding features. Yes, they could die from a broken code deployment to production servers. But they'll also die if Myspace or Google+ is more aggressive in adding social networking features. Being ultra-conservative just to "avoid breaking things" will mean the business can go bankrupt and thus, there's no website to even worry about breaking.
[1] excerpt from: https://theweek.com/articles/462863/stories-behind-3-great-b...
MFaBT is a great strategy for design prototyping, not engineering. In engineering, you have 18-24 months to get a product out the door: there are no prototypes. In design thinking, the timelines are different, and you expect the first 20 versions not to work.
The academic attitude is appropriate for critical systems and to a lesser extent for systems that can't get realtime updates, but if your app is non-critical (especially entertainment apps a la social media) and you can roll out a fix in a couple of hours, moving fast and breaking things is perfectly fine, especially for features which don't constitute core functionality. In the saas world, bugs are typically very cheap (or can be made very cheap via continuous deployment, microservice architectures, etc) and opportunity costs are through the roof. In that sense, moving fast and breaking things is probably eminently desirable, since it's cheaper to react to a bug than to delay an opportunity because you want to do quality assurance as though your software updates need to be mailed to users on physical media.
Sure, but the difference is scope of impact. If you want to learn to ride a bike, ski, write a book, be good at sports, whatever, then you're going to make a lot of mistakes in practice, where they won't matter, where they won't be published, and where they won't affect users. In MFABT, production isn't sacred, and that's the problem.
I have a problem with my coworkers making the claim that it is actually safer to move fast. If you're uncertain about your software, release it now, rather than later because you'll become even more uncertain as time goes on.
If you have the ability to ship things fast - and so do regularly - you will find that your releases:
- Have a small blast radius if there are bugs, because how much blast can you have in an hour's work versus a quarter's?
- The tooling that you build to let you go fast will tell you that a blast is indeed happening right now
- Your Mean-Time-To-Resolution (MTTR) is lower because you can just roll it back. It's not days or weeks of work to get a rollback out, it's minutes.
- That having a culture of being able to release continuously and quickly allows you to go fast, and the occasional breakage is tolerable, because the cost is low.
You can go slow if you want. And if you don't have LEST metrics or have slow release processes you must, but if you work to get to the point where you are able to move fast, you'll find that you break things less and fix them quicker when they do break.
The alternative is you still break things, but the problems incurred are much, much greater.
Also note: there are some environments where continuous deployment is never going to be an option or desirable. Thankfully these environments tend to be ones where the market will tolerate the higher costs of formal methods and a more deliberative quality control process (medical devices, aerospace, nuclear reactors, etc.).
If you have a shorter period of time to produce a smaller change, are you more or less likely to have a smaller blast radius than if you have a longer period of time to produce a much larger change?
There are no absolutes, but we're talking about risk here, and risk is about impact and _likelihood_. Your impacts are going to be smaller and your likelihood of big blast radiuses are _less likely_, but not ruled out.
Having said that - a smaller piece of code is easier to understand, so there is more potential to notice problems.
However that depends on whether you do that diligence. And the dangerously false belief that the blast radius is going to be smaller because of change size rather than understanding works against you.
It’s a red herring. Risk is lowered because the code is understood better not because the code is smaller.
The assumption that each change carries an equal risk of disaster isn’t valid, because risk depends on your understanding of the changes and the codebase.
Risk is equal only in the case where the developers know nothing about the change or the codebase. In this case, fire the VP of engineering or equivalent.
Making changes as a series of small increments can just as easily lead to less understanding of the overall impact of the changes than a larger design process. Then you get a nice big failure from a small change. It looks like most of the changes were harmless when in fact they were all just building towards the one that caused the failure.
bug size * mean time to recovery
So if you release early, release often and keep an eye on your metrics, you stand a good chance of your blast radius being consistently low.
You won’t eliminate it, you’ll just be beating the team with rare and large releases that take an age to get through their release process.
Of course, if you can take your time and make small changes slowly, but release them quickly, you’re now doing even better, but there are few businesses where that’s economically viable: competitors will just get to product market fit quicker than you will.
What is the underlying mechanism that makes it so?
Imagine if doctors, engineers, plumbers, electricians, and all the other real world folks we depend on decided this was a good way to operate. Would you drive across a bridge knowing that was the motto of the people who designed and built it?
Surgeons must act fast and be good at multitasking. I bet they are not in a forum discussing if those traits make them vulnerable to hasty decisions and to loss of concentration on the task at hand.
Competent developers, on the other hand, have extreme trust on the judgment of their peers. They know they won't break the production build at will; if they ever do, they will use all information gathered to improve the team's infrastructure and their own development process. They are not afraid of using the term, because breaking things is the most valuable thing that you can do for your learning, but, most importantly, they realize that other people's lives and goods are potentially at stake, and act with due diligence.
So, "break things" is not an excuse to break the production build at will and move on to the next task. If competent developers ever find themselves in a team that does so, they try to educate the team and, if that does not work, GTFO.
Just to make things clearer, I am a programmer myself, and have never been in a management position before. My overall impression is that I learn faster when I make mistakes and need to fix them. Expecting otherwise would contradict most of the experts on human learning.
Worst analogy ever. If we had GIT for woodworking you would see productivity skyrocket. But you can't restore wood like that. You can't restore pipes, wires, structures like that.
Code.. you can. So get out of here with your analogy. I can move at a very fast paced coding new ideas at times, or implementing random stuff knowing I will probably break stuff. But you know what? I can fix that afterwards. It doesn't matter that I broke something to get something else to work.
This process helps me in so many different ways, and seems so productive, but I’ve never found anybody that uses such an approach. I would love to discuss it further with other people so that I can improve it.
Move Fast and Break Things always meant “and don’t be afraid that you might accidentally break things, because you eventually will and you should just fix them if you do”.
The saying isn’t itself a fallacy but doing the moving fast part without adequate resources to do the fixing part is indeed reckless.
Don’t push on a Friday night. Don’t push without metrics on error rates. Talk to your team. Append instead of overwrite. ..etc.
Sometimes, teams get stuck in 'analysis paralysis' -- and this was meant to be a type of antidote.
To the folks that say this is a type of anti-pattern, I say: Either you haven't seen this implemented correctly (it helps if you have kids that you want to encourage to explore) or "There is more than one right way to be creative".
Move slow, break things anyway, disable tests to hide it.
Yeah, I've quit.
Ha, amateurs - my last company never wrote any tests in the first place.
When you are still prototyping and have not found product-market fit, it might work. (And that quote comes from the early days of Facebook, when it was still morphing into its final form.)
Once you have established yourself as a viable product, found your fit, and have paying customers, "break things" becomes a path to poor service.
Yes. I have seen this too many times. People are creatures of habit. I have seen people from small companies try to run after the growth a company 10x or 50x bigger like if nothing has changed. I have also heard of, not seen by myself, directives from big companies going to smaller ones trying to apply all their processes and checks to just a couple of teams.
Each strategy is correct for the company situation. There are no silver bullets.
Friendster and MySpace by contrast slowed way down once they hit traction, but it didn’t enable them to solve their scalability issues.
do we live in the same universe ? facebook has already one full step in the big storage room with myspace, omegle, etc. most teens I know laugh at you if you say you're using facebook like you're some decrepit old person.
I think a better one would be "don't run your company based on aphorisms".
People act as if because "Move fast" is a good thing, it has anything to do with "Break things". Does everyone at facebook sprint around the office and break limbs when they bump into each other around a blind corner? No, because they're not morons.
A "cowboy" mentality may indeed help a start-up edge out competitors. But applying that to an established company can ruin your reputation: you have more to lose by playing fast and loose. Startups by nature are a gamble, but if an established corporation gambles too often they may poke a hole in their dam that they can't patch fast enough.
Venture capitalists usually factor in a high failure rate to startups they invest in. Their plan is that enough will succeed in a big way to counter the high failure rate.
A lot of "fads gone wrong" happen when a technique or technology that works for one situation or size is misapplied to another. Your office printer inventory tracker doesn't need microservices and web-scale databases.
Pilot projects with new technology is often a good thing, but don't bet key organizational functions on such a project (sometimes called line-of-business systems).
> Yes, elite teams ship faster. But, their changes are 1/7 as likely to fail, and they recover from incidents 2,604 times faster than low-performing teams.
Since the report is behind a form, and the form is having trouble loading for me right now, I’ll just ask if anyone knows the context behind these numbers?
These numbers seem very large for any kind of average; are they choosing the most extreme cases? I feel like they also only make sense in units of seconds or minutes, otherwise we’re comparing hours to weeks or days to years.
So quite often, I guess.
I do think "move fast and break things" has a place, but that place is mostly limited to so-called transcendental problems: situations where the status quo don't work and we need to try something outside of the established solution space. It's kind of comparable to the over-use of brainstorming; quite often brainstorming is not going to help you find the solution to a problem.
This is meant to include changeability: quality code should theoretically yield faster iterations too. This “break things” aspect is something I believe he opposes.
Is he right? Wrong?
As to MFABT, the proper adjustment of that knob in a particular situation is difficult and is what expertise is all about.
"As consumers and end-users of software, our expectations are continually rising."
A very agreeable statement, but no they're not. Expectations are that your service works, makes sense, is easy to use. This is not a rise in expectations, it has always been there.
What are the consequences when you try to move fast and not just break things, but fail?
- It takes longer to resolve incidents.
- You lose customer confidence and sales.
- Employees burn out, and you have a high turnover rate.
- Taking longer to resolve incidents will allow you to anticipate minor issues that could add up during a worst-case scenario, which is a valuable thing.
- Impact on customer confidence is less critical in earlier stages, when you are not expected to make a profit. Once again, these minor impacts on it will, ideally, bring attention to things that matter.
- Employee burn out is uncorrelated to this approach. It is all about having the right policies in place, replacing blaming by accountability and treating other people with due respect.
* We don’t know what the final solution will look like, because of things like scope creep and client demands.
* The more agile the team is, the higher the velocity and ability to iterate.
* The more specialized roles we have (like project manager, team lead and junior developer) the more we can focus on completing tickets to increase velocity.
Which I counter with:
* We actually do know what the final solution will look like. I normally have an idea of what clients are asking for before they finish their sentences. Then I extrapolate to the fundamental problems and meta concepts around their specific problem. I’ve already formulated solutions and am solving them in my mind, until I isolate the unknown steps (the hard steps), whose odds of me being able to solve them are the basis of my estimate. Most of my work today goes into research, setup, glue code, infrastructure and maintenance. Solving the client’s actual problem constitutes perhaps 5% of the work I must do.
* We can always iterate faster in our minds than in the real world. With a proper architecture budget, we can take time in the beginning to brainstorm the complete solution before we ever start writing code. Then if we follow proper quality assurance practices like test-driven development, we can write code that is free of technical debt and bugs. All the while exerting substantially less effort than writing the same thing several times.
* Agile programming wasn’t really a thing until the rise of large internet companies and the marginalization of individual software developers. It’s curious to me that the people with the most say in writing solutions have the least clout in organizations. I feel that this is due to supply-side economics and wealth inequality. So today we cater to client first, then organization, then software developer. That’s what the specialization in agile is really all about. But we should be doing something more like movie production, so the client is the producer, the organization is the director (or movie studio) and developers are the actors. The industry should have more confidence in the process and have conventions which mitigate the churn caused by clients’ fickleness and low understanding of the problem domain. But right now it feels like the emphasis is on getting every scene filmed as fast as possible. We pretend that developers have no gravitas and are just interchangeable cogs in the machine. It’s no surprise to me that most productions bomb in the box office.
To use construction as an analogy: programmers used to be akin to architects, spending time in the beginning to draft a plan that includes all contingencies so that the building doesn’t fall down (a right-brained or politically-blue approach based on creativity, insight, seeing outside the box, etc). Today we’re being told to be framers, hammering out solutions as fast as we can over and over again in order to get things done until the building is finished (a left-brained or politically-red approach based on a strong work ethic, discipline, loyalty, not getting distracted, etc).
I think these are both important, and over the last year I have been working on my “getting things done” side to improve my execution ability. But I will always be a dreamer and value the intuitive side of software development. The idea that we can manifest solutions in the real world with the power of our minds is what got me into programming in the first place. The trend towards application and turning programming into just another job has soured me on the field in general. I’m actually not entirely certain that I can do it anymore. So to me, the true cost of move fast and break things is the loss of people like me who thought that this was all going to go a different direction.
To me it further confirms it - don't trust either the industry, or the people in it.