Lessons from a decade of research on how tech startups succeed long term | Hacker News Reader

Lessons from a decade of research on how tech startups succeed long term

jasonshen.com

70 pointsjasonshen11 comments

There are great insights that can be learned through these kinds of analyses, but they can also be problematical, suffering from a severe selection (“survivorship”) bias and confusing correlation with causality. To do a scientific study would require picking some attributes thought to lead to “greatness”, rating a random selection of companies along those attributes, and then tracking their outcomes going forward. Feynman had a great essay about these kinds of problems, “Cargo Cult Science”, which you should go read right now if you never have http://www.lhup.edu/~DSIMANEK/cargocul.htm

For example, the companies featured in Jim Collins’s books haven’t fared especially well in the years after those books were written. Cf. http://www.boston.com/bostonglobe/ideas/articles/2009/04/12/... http://thehaloeffect.typepad.com/authors_blog/2007/02/jim_co... http://bobsutton.typepad.com/my_weblog/2009/04/a-wellcrafted... http://hbr.org/2009/04/are-great-companies-just-lucky/ar/1

That's a great thing to note. Correlation definitely does not indicate causation. In a perfect world, we'd have one group of companies employ the Great by Choice practices, and in another we employ a different set of practices. However, that's not really possible.

The point about the other companies is less relevant because companies do change and stop doing all the things that make the great. The Good to Great companies had 15+ years of sustained success - doesn't mean they will be amazing forever.

Similarly, just because Arnold Schwarzenegger isn't as ripped as he used to be doesn't mean that lifting heavy weights and eating a lot won't make you gain muscle.

Yeah, but the point is that “success” depends on a huge number of factors, and a post-hoc ethnographic analysis isn’t necessarily going to pick out what the real differences were. It’s quite easy to fool ourselves, as Feynman describes.

Even if you had an economy where all companies were managed identically, the sheer number of companies would lead some of them to be successful for 15 years in a row, and others to go out of business right away. Looking at past history and trying to sift through it for patterns is a great way to form hypotheses and questions for further research, but it’s entirely unscientific to claim those patterns are causal without setting up a new experiment.

To take your Schwarzenegger example, we can tell that lifting weights and taking steroids builds muscles because we’ve run the experiment where we take two otherwise identical groups of people, and give one steroids, and then observe the difference going forward. If we only look post-hoc, we might come up with entirely irrelevant patterns, for example finding differences in the amount of time successful bodybuilders spend socializing or watching TV or the number of herbal supplements they take or the kinds of houses they live in, &c. when compared to the general public.

Anyhow, I’m not arguing that we shouldn’t try to do ethnographic and historical analyses (my father is an anthropologist/historian!), only that we need to be cautious and skeptical of our conclusions.

Yes, if you've read "The Halo Effect", which demolished much of "Good to Great", you'll want to wait for whatever book demolishes "Great by Choice" before reading it too.

Decade-old business books are SO much more enlightening; you get to see how history really turned out.

The other important thing to note is that:

They didn't just study success, they studied success AGAINST mediocrity

These findings aren't just what the 10x companies did. It's what the 10x companies did that the mediocre ones - which were very similar in size, product, team, market - did NOT do or vice versa. They also only selected practices that were done for nearly the entire time studied, in the majority of the companies studied, and that look to be reasonable explanations for the different outcomes. We're not just injecting data into an algorithm.

And to respond to jaylevitt - thanks for mentioning Halo Effect. I really should find the time to review it as well.

Yes, but “pick a smallish set of successful companies and then try to find one other vaguely similar company to each for comparison” is not enough of a control to guard against these kinds of biases. I understand that experiment design is really hard when dealing with something as big and complex as public companies, and Collins does much better than most business books (just by trying at all). But this kind of experiment design would never fly in any hard science.

Thou captured much of my reaction. 9 years doesn't make research rigorous if the methodology is flawed, and the end sample sizes are small. Comparison companies are good but don't solve methodology concerns. And I am actually one who lIkes Jim Collins. It's just bad to view it as rigorous.

Great (and extremely thorough) summary of Great By Choice - a book I think everyone building sustainable businesses should read. While a lot of the points Collins makes will be familiar to those familiar with the Lean Startup, GBC provides some awesome, real-world examples of how to translate those "lean principles" into action.

However, I think you gave one of the most valuable and counter-intuitive points in the book too light a treatment: That successful 10X'ers not only consistently hit their goals, but actively tried not to exceed them.

Living in the start up echo chamber you'd think that "speed" was everything; we do "agile" development, we take venture rounds to "supercharge growth". Hockey stick graphs are treated like the tech world's pornography, depicting circumstances just a tad too good to be real. The mantra is that if a little growth is good, a lot of growth is really good.

GBC turns that on its head. Growth is good, but only a little bit. Only as much as you can handle. Explosive growth can quickly become uncontained growth can quickly become webvan. Facebook limited itself to college campuses, taking years to go public. Southwest started with 3 planes and didn't expand their routes until they had to. Zipcar stayed in Boston until it was absolutely sure it was ready to go to NYC.

The 20 mile march is as much about having discipline in self-restraint as it is about pushing forward.

That's a great point Ryan and I'm glad you brought it up. Everyone chases after "hockey stick growth" but it's important to note the cost of such growth: challenges with scale, whether that's with the codebase, server infrastructure, hiring, increased spending, etc. Example: Twitter. The fail whales almost brought them down.

I think this is partially driven by investors funding things that have that hockey stick so hopefully some of them also read GBC and adjust their approach accordingly.

I won't hold my breath on that one though =)

Jason, do you find the MoodThingy useful? and if yes, in what way?

You are the only one I know who uses it so I am curious to hear your experience.

Hey aymeric - I do like MoodyThing. Feedback is always valuable to a writer and I adjust my posts based on the comments and emails I get, so this is great, low-friction way for me to understand what people get out of my writing.

Not sure how much it slows down my site though - and whether that trade off is worth it. =)

Reply on news.ycombinator.com