Killing the Crunch Mode Antipattern (2014)
chadfowler.com
chadfowler.com
The film industry has most of the problems of software development and game programming. But because Hollywood is heavily unionized and overtime is paid at 1.5 to 2x normal, film scheduling and budgeting is a well-developed discipline.
Film production has something known as a "completion bond". This is an insurance policy that guarantees to the investors that the film will be completed. The insurance company has the right to fire the director and anybody else and take over the production if it goes significantly over budget. The result may not be great, but you will get a movie.
"Bad Girls", a 1994 Western [1] is an example of a movie where the insurance company stepped in and took over. They put in a new director, who spent a day looking over the production and talked to everyone. She then sent the camera and lighting crew back to Hollywood, told the set builders to finish the two sets they nearly had finished and forget the others, put the stunt director on teaching the lead actresses some horse riding tricks, told the costume directors to come up with some bad-girl riding outfits, and then went off with the writers to hammer together a script that used those components. In a few weeks, they had a movie. Not a great one, but one that made back $15 million, which is a lot better than zero.
That's how you beat the crunch problem. Underestimation in the film industry applies great pain to management, not the employees.
Sounds like market-based regulation. It would be interesting to see an example contract with the insurance clauses and list of conditions which trigger a change of direction/control. Are there parallel conditions in VC term sheets? Are there periodic reviews to determine whether the project is on track, or does the insurance company only get called in when the project approaches its budget? Any creative industry would want to avoid micro-management by insurance risk managers.
Here's one.[1] There are three parties, the producer, the financier (the party putting up the money for the film), and the guarantor (the insurance company). The contract calls for a "strike price" (the maximum film budget) and a delivery date. If the producer and director overrun those, the financier has the option to invoke the completion bond. The completion bond company can then 1) take over, or 2) abandon the picture and pay the amount of the bond to the financier. If the completion bond company takes over, they may have to put money into the production, in which case they get all gross receipts until they are paid back.
A well known director with a track record might get terms like that. Further down the food chain, the terms get tougher.[2] The bonding company watches progress and spending closely, and if the schedule starts to slip, sends observers in to monitor the production, and may take over before the production runs through all the money.
There's an up-front phase before production, during which the guarantor reviews the production budget, script, and shooting schedule for realism. This is where the real action happens. The guarantor wants to prevent cost and schedule overruns, not take over production. Completion bond companies are good at cost estimation; they see the full cost data for hundreds of films and know what everything costs. If the producer's estimate is, in the opinion of the guarantor, likely to result in an overrun, there are negotiations between the three parties. Either more money or more schedule has to go in, or some expensive or slow scenes have to come out.
This third-party check, by an independent party who's on the hook if there's an overrun, forces realistic budget and schedule estimates.
A completion bond costs 2% to 3% of the total budget.
If software development (especially game development) had that, there would be far fewer "crunches". Remember, in a unionized industry, crunches and 1.5x to 2x overtime pay increase the cost per unit of work.
[1] http://www.filmfinances.com/pdf/Completion%20Guaranty.pdf [2] http://www.eqgroup.com/completion_bond.htm
Thanks for the insight and sample contract. Many large software projects could benefit from formal benchmarking of estimates against a history of success/failure. Challenges would include cultural pushback, normalization of estimates across different subject domains and corporate concern about trade secrets and IP.
>The bonding company watches progress and spending closely, and if the schedule starts to slip, sends observers in to monitor the production,
These observers/PMs must be highly competent and well compensated, in order to sustain relationships with multiple creative teams while adding value to an enforcement role. Analogous roles in software development might be VC technical due diligence or PM supplied by a prime contractor on a large project with various sub-contractors.
Okay, but how would this work in the software field? If a product is significantly over budget, that usually means the software has a ton of unfinished features and bugs. You can't just bring an outside project manager or whatever and expect them to set things right in a few weeks. There's a lot that goes into making movies to be sure, but making software is arguably much harder.
Also, there managers who can take over a failing software project and right it. But since software is so unruly, any sensible bond company would set the budget threehold early enough to prevent those problems.
Lead: "How long do you think this will take you?"
Me: "Two months"
Lead: "I already promised the customer we would have it to them in 3 weeks."
Actually meeting a deadline happened so rarely I'm surprised people weren't constantly getting fired (no one got fired that I know of. Well, except our CFO for some undisclosed malfeasance). In reality, people just got burned out and quit. One especially smart guy left after only a month on the project.
After my first few failed attempts to meet an impossible deadline, I just gave up trying. I worked a straight 8-hours and left at the end of the day, waving at the missed deadlines as they sailed by. Still couldn't get fired. Damn, guess I'll have to quit. Got a new job for a small pay cut (though in a lower COL city).
I don't mind the very occasional crunch. Stuff happens. But my tolerance level for "constant crisis mode" BS is just about nil nowadays. I must be getting old.
There's a great quote for this: "Bad planning on your part does not constitute an emergency on my part".
There's the problem. Developer performance was being measured according to a framework in which it was likely to fall short. This is because for many problems software development is inherently exploratory and difficult to spec out up front in sufficient detail to know in advance where the surprises are. If one is worried about being fired for not meeting deadlines, one should probably quit as soon as possible and find better work.
Management never grasped how poorly things were handled on their end. I grew to resent the company and its leaders, and had no choice but to move on.
IMO, that's the whole point of constant crunch mode. Everyone is a hero, which makes it more difficult to call out dumb decisions. In the places that I've worked at that abused crunch time, the management was over its head completely and was unable to get responsibility delegated to appropriate places. So the VP or even C-level execs were making decisions about LUNs (that they were unlikely to be qualified to make)
It's easier to congratulate the heroes than to actually manage.
In the worst environment, senior leadership patted themselves on the back for having as many as two dozen people on incident response calls in 15 minutes. Problems didn't get fixed, but by golly we spread the suffering around!
In the Capability Maturity Model 'Crunch Mode' is Level 1 - Initial (Chaotic).
Also, learning to say "no" is vital to preventing crunch mode. Wanting to please my superiors, I would typically say "yes, I'll do my best" or "maybe" when things were dire but never "no, that's not possible".
In short, you can use crunch time strategically to meet deadlines but you always need a recovery period afterwards. If you work 60 hour weeks, in about 4 weeks your productivity will drop lower than it was when you were working 40 hour weeks - despite putting in 50% more hours. Also many people in this burnout zone will self report that their productivity is higher than it was (and they're wrong). If you've been crunching for a month straight, you're working ineffectively and you're too tired to tell.
Here's a presentation on the topic from Dan Cook, with links to papers: http://lostgarden.com/Rules%20of%20Productivity.pdf
And here's a fantastic write-up of a recent quantitative study in the games industry. They looked at how the success of video games correlates with crunch time and overwork. I suspect that these results would also hold true amongst startups: http://www.gamasutra.com/blogs/PaulTozour/20150120/234443/Th...
Crunch time in business is a symptom of poor time management by management.
True, the author does not provide empirical data, but anyone who has worked through a few projects culminating in a rush is familiar with the effects on person and quality.
I ended up being quite prescient - the project took 3 1/2 months, including a bad crunch near the end of it. Fortunately I was placed in a different team before that crunch, I probably would have quit fast after surviving an insane crunch a few months beforehand where I wrote ~50% of the frontend code due to a series of unfortunate circumstances.
Crunch mode makes developers want to quit and distrust managers/executives - it's a breach of trust.
1) You think the work your team doing is already of such low quality that it can't get worse, and increased "quantity" of low quality work is better than the usual.
2) You've created a crisis and feel you can convince your team that this is just the natural state of capitalism, or that by accepting salary they are implicitly accepting the debt and promises that you made to your VC firm.
3) Feature lists and time to market trump actual performance. You're trying to get something that can be demoed and sold without regard to how well it actually works. Maybe your product is for a niche industry and you can make a convincing business case on numbers alone, or maybe the people who do the buying are not the people who will use the software.
4) Your product is on shifty legal footing. Having a saleable product will bring you legal support, but until then you are a sitting duck for a lawsuit or regulatory action.
5) Your team is capable of better work, but you believe that the only way to extract it is to create a crisis.
A few weeks ago, my startup[1] did a crunch mode week before releasing our beta to the first set of users. We had promised access to the application on a certain day, and I am absolutely never going to be convinced that missing a deadline that you've promised someone else is an okay thing to do, as the article suggests as a form of self-punishment. In doing so, I did a few things in order to take care of my one employee. First, I said that lunch and dinner would be covered by the company, since we were both staying there until the nighttime hours. Second, I made a Costco run and got us anything would could need to snack on or drink during the week (not very expensive when you're buying for two people). Third, immediately following our successful launch on a Monday, he had Tuesday and Friday off. It's very important that when you're not on crunch that it feels like you're not on crunch, and you "make up" for the time that you burned through.
I am always of the opinion that if you're not willing to put in a ton of hours, there's someone just as smart and focused as you that is. I think my perspective has been forever changed, though, because I've been on deployments where we worked 12 hours a day for 9 months straight with no holidays, doing knowledge work I might add. Becoming accustomed to that makes everything that comes after it seem incredibly easy.
Both my employee and I feel incredibly lucky to be working on something that we both really believe has a chance to succeed greatly, and it is so much a part of our lives that when we're not at work we're thinking of more ways we can improve our product. When you're that deep into an idea, be it a business, hobby, love interest, etc., the amount of time you spend on it always seems like not enough. I think people bristling about long work hours are doing so because they haven't found something that makes them feel this way.
Or they are not in a position where they stand to profit in proportion to the extra work. Extra time with out extra compensation means the employee's individual ROI drops, and that is not something a smart person should accept on an ongoing basis.
Or maybe they just don't like being forced to work for free. What you're saying is that employees should be glad that they get a chance to work overtime for free. They should be glad to be victims in other words. That's just insane.
Mind you, another wise man once said "there's a sucker born every minute", and psychologists have identified Stockholm Syndrome as being a thing, so he could be right.
That will happen even if you pay employees overtime, which most companies and proponents of overtime are vehemently against because the real reason they demand it is because they want to exploit free labor along the lines of temporary slavery. No wonder there is so much shitty code out there, especially at startups and other companies that love to overwork their employees without paying them. Then they call it "technical debt," as if there was anything technical about depriving employees of sleep, health, and their life outside the company. And finally, to top it all off, they want us, the employees, to enjoy being worked overtime and having our health ruined and our time stolen. The delusions of management and executives in this area is incredible.
But I don't think it's fair to call employees willing to subject themselves to this suckers, only the ones who do so knowingly and willingly. Many people simply don't have other choices or have other obligations (family, mortgage) that make changing jobs difficult. Others, especially young people, just don't realize what they're doing to themselves until they burn out or are too caught up in the social "coolness" of it.
"Fire lit under your ass" doesn't mean "work 16 hours a day." If you want to succeed, you should be focused on producing and maximizing the right outputs, not measuring time worked, which is simply an input.
> I am always of the opinion that if you're not willing to put in a ton of hours, there's someone just as smart and focused at you that is.
That might be true, but can you recruit and retain those smart and focused people? In today's market, the talented and experienced have options. Unless you're offering top-notch salary and benefits, chances are you're not going to be able to retain and recruit the best people while demanding they live in a constant crunch mode. They don't have to subject themselves to this to earn a good living.
> Both my employee and I feel incredibly lucky to be working on something that we both really believe has a chance to succeed greatly, and it is so much a part of our lives that when we're not at work we're thinking of more ways we can improve our product. When you're that deep into an idea, be it a business, hobby, love interest, etc., the amount of time you spend on it always seems like not enough. I think people bristling about long work hours are doing so because they haven't found something that makes them feel this way.
Is it not possible that people bristling about long work hours are doing so because they have other interests in life besides work?
While your earliest employees should drink a little bit of kool aid, as you scale a company you can't reasonably expect your employees to be as invested as you are. And you should be mindful of the fact that the people who provide the level of dedication you seem to be looking for are often the ones who burn out and become the most disillusioned and resentful when you hit the inevitable bumps in the road.
> That might be true, but can you recruit and retain those smart and focused people?
There's also the problem that after a month or two of 60-hour weeks, that person will be less productive than someone who works 40-50 every week and saves emergency mode for the rare genuine emergency. So even if you can attract and retain that person, you'll be overpaying for shoddy work. It's a wonderful economic outcome, in which both parties are suckers.
You did -one week- of crunch. That's not what the article is speaking to. You're talking that "if you're not willing to put in a ton of hours", "my employee and I feel incredibly lucky", etc etc etc, but you only pulled a -week- of crunch. This isn't talking about where your down time outside of work sometimes incorporates thoughts of "what could we do better at work?", nor is it talking about "We're so close; if we just work a little longer this week we'll hit this deadline, learn from it, and can take a break right after", it's talking about the repeated pattern in some places of crunch mode all the time, > 40 hours a week for multiple weeks, which research has shown will lead to negative performance.
Joking aside, I have only sold 40h of my week, not because I wouldn't be happy to sell you more hours but because the rest are already sold.
No one is suggesting that you just miss deadlines. The suggestion is that you get better at setting deadlines.
You may be living and breathing your exciting project but, really, how many lives were saved by setting such an aggressive deadline? How many businesses did you save from bankruptcy? If your own, then you might want to think harder about saner financial planning as well.
Very few projects need to rush to the finish line to be successful.
You live and breathe an exciting project. I get it. But how is the quality of your thought process affected by pushing yourself to the limit everyday? People need space to think. People need space to see outside of their obsession. Not allowing for that misses opportunities or, worse, puts you in a position to be surprised by something outside of that obsession.
The rest of us aren't stupid. Many of us are seasoned. We realize that one must be wary of obsession because keeping your mind in one mode for too long makes it very hard to move from that mode or even eventually realize that other modes exist. And, make no mistake, eventually reality will force a change. The question is, will you be cultivating the flexibility to deal well with it or will you be stuck in one mode?
You're running a marathon, not a sprint. Let your demons drive you but make sure you can take the reins back when you need to.
1) how many extra hours did your employee work (i.e. over an 8 hour work day)?
2) if extra hours > 16, how do you feel about that? (FYI as an employee I'm pretty eye-rolly if I get a day off as a "favor" after burning >=8 hours in "over time" -- the = is important!)
http://www.simplethingcalledlife.com/2015/elon-musk-usc-succ...
> And if you do the simple math, say that someone else is working 50 hours and you’re working 100, you’ll get twice as much done in the course of a year as the other company.” – Elon Musk
Elon is a typical SV guy right?
It does seem as if he's very smart and able to lead significant enterprises to success, however.
There might be two takeaways here:
1) Even smart/successful people may believe things that are wrong.
2) The point of Musk's statement may be something other than to communicate an accurate model of output vs labor.
Meaning, taking comfortable estimates (on the order of 1-2 weeks) and just, by default, sliding them earlier by one or two days?
It's a subtle way for management to cope with the anxiety that is trickling downhill, without engaging in or admitting a period of Crunch Time.
Management: "How long will it take?" Me: "N weeks" Management: "Ok, you have N/2 weeks"
I've been told they actually teach this at business school.
If that gets you N/4 weeks to do Y, or still N/2 weeks to do X, get new management.
Business school is just expensive signalling.