Lead Bullets (2011)
bhorowitz.com
bhorowitz.com
"You must never confuse faith that you will prevail in the end, which you can never afford to lose, with the discipline to confront the most brutal facts of your current reality, whatever they might be."
The customers were buying;
they just weren’t buying our product.
This was not a time to pivot.
Because his post goes like "guy says lets use lead bullets." -> "We hunker down and use lead bullets" -> magic -> "Boom. $1.6 billion company"
Optimize something, reduce the running time by a few ms, optimize something else, reduce by some more ms, repeat 8 hours a day, with several people in paralel for a few months.
The concept of a "lead bullet" implies that the history won't be interesting to read, just some repetitive boring hard work.
I may be the only one, but knowing what the 'something' was is incredibly valuable bc we all, as people, pick 'something' and not everything in any sort of bullet is created equal.
- Processes which run many times.
- Processes which run long time (and block other processing).
- If at all possible: both.
The more subtle approach is to identify things you don't need to do (needless initializations, "settle" delays, doing (or allowing to be done) the _wrong_ thing requiring both unwinding that operation _and_ doing the right one, and anything you don't have to wait for (if it's an ancillary process, spin it off and handle it out-of-sequence). Redesigning a process to avoid sorting data. Eliminating global locks. Spinning off tasks to parallelized processing (especially if you have multiple cores/systems available. Pipelining data to avoid writing/reading from disk multiple times. Handling data in memory rather than on disk. Realizing you _can_ handle data in-memory, and that disk-seeks will kill you, so sorting or bucketing output. Swapping spinning rust for SSD. Specific tools such as COW, snapshots, or the like can be very useful in such cases.
Algorithms (or methods invoking algorithms) can be another big win.
I've seen processes fall in time requirements by 90% following such an optimization process.
1. Use a sampling profiler on the important execution scenarios.
2. Identify the operation that is taking the most time.
3. Ensure it is using appropriate algorithms and data structures.
4. Profile again; if it is still taking the most time, hand-optimize it.
5. Goto 1Even if it was boring or rote work, I'd love to know what it was.
Too many people are unwilling to do the hard thing. Probably because their "goal [is] business for the sake of business (http://al3x.net/2013/05/23/letter-to-a-young-programmer.html)
The way I'm reading it, the point is to concentrate on your product and making it, in reality, better than the product(s) of your competitor(s) rather than... pivoting? Building a different product? Making your product into something else? Changing your target audience? I guess the only real lesson I'm taking away here is that "if you have product, sell it; if it sucks, make it better." Not exactly groundbreaking, though certainly true.
If the desire was to communicate something more general, taking it out of the context of existing product-based businesses with market share and competitors would probably be necessary (but would have probably killed the anecdotes... which is why I'm guessing the article didn't really get off the ground in this regard).
So yeah, it's mostly true... but where it is, it's sort of trivially so.
Instead, you often know what you need to do - it's a long, hard slog, and it's about sticking at it longer than your competitors.
It's a recurring theme in Ben's blog:
>> Mediocre CEOs point to their brilliant strategic moves or their intuitive business sense or a variety of other self-congratulatory explanations. The great CEOs tend to be remarkably consistent in their answers. They all say: “I didn’t quit.”
http://bhorowitz.com/2011/04/01/what%E2%80%99s-the-most-diff...
There is also the opposite problem (one much more common in newbie founders IMO) -- seing lead bullets everywhere when a silver bullet would do. I used to believe I can just outthink and outwork other people by sheer force of will. I still believe I can, but I would have saved myself (and my team) a lot of effort if I learned to look for silver bullets when they fit the problem.
Telling lead from silver requires some learnin'.
You need to have some realistic assessment of how badly you're losing. Gungho "Yay effort!" is not a worthwhile risk analysis.
If you look at the past decade and compare the technologies that "won", you will see that in almost every case, they were faster than the alternatives, and had performance as a major selling point. The notable exception is Ruby on Rails, but it was sold so well (the original Rails screencast was ground-breaking) and it solved such a huge pain point (XML hell) that, in this case, it didn't matter (though even they had to answer the question "but does it scale?" at some point).
Sometimes hookers, expensive lunches, tickets to sporting events and being golf buddies with managers are what counts. And if those are what counts, then those are the lead bullets.
One silver bullet (E = m c^2 and Einstein's letter to Roosevelt) and some lead bullets (the long slog of building huge industrial plants to purify the uranium). Maybe sometimes you need both.
I wonder if this story is related:
"Thou shalt execute".
Ben is saying that at times need to work very hard? Okay.
Ben is saying that if a competitor has a much better product that is selling well and threatening own business, then likely just must do the work needed for a competitive product. Okay.
But after that I lose it with Ben: His "lead bullets" are defeatist, close to luddite, look like he is trying to degrade himself and his company, to fight by getting down, dirty, simplistic, nose to the grindstone, shoulder to the wheel, and ear to the ground and trying to be successful in that position, to suffer first and count on that being the path to success. Not good. Indeed, one of the things we're supposed to be doing is innovating, which likely also means being creative, original, maybe even mathematical or scientific.
Ben reminds me of Edison sending yet another team to the Amazon jungle to look for more plants that he could bake into carbon and try for a light bulb filament that wouldn't burn out so soon. The real solution was to borrow the idea of a tungsten filament from the chemist Swan in England and to use a really good vacuum pump from a guy in Germany. Edison could have had his teams slogging through jungles for 1000 years without finding anything. But, eventually Edison did go with tungsten and the German vacuum pump.
For Windows' first version of IIS being as much as five times faster, tough to believe that just a 'lead bullet' approach would yield what was missing from what Ben had. Instead, look at the architecture and think a little about what the heck the poor computer is being asked to do. Then look at where the computer resources are going. Likely then, tweak the architecture: So, if are spending time walking down arrays or linked lists in O(n), then consider using an AVL tree in O( ln(n) ). If executing the code for a page when know that the output will be the same as the last 10 times, then cache the output and don't run the code again. If caching the output in main memory, i.e., virtual memory with page faults, then consider just using disk file for each page to be cached and save on page faults. If are connecting to a database, then have a pool of connections. Etc.
I.e., think a little. Looks like Ben doesn't like thinking, more likely has, really, a low regard for his people and doesn't trust them to be productive if asked to think. Ben wants to measure hard work by sweat, long hours, and suffering and not good work with good results.
Here's a lesson I draw: I agree with Ben that a business that has been going along well might suddenly encounter some problems, e.g., a five times faster competitive product given away for free.
First, such problems should have been expected. If Ben had inefficient software, then he should have known that and not had to learn it from a Microsoft product. Or, how and why did the Microsoft people do a much better job? Sure: They wanted something fast, at least not wildly inefficient, thought some about the architecture, and did a decently efficient implementation. Why? Because they cared enough. Then before Microsoft's work, Ben hadn't cared enough.
Second, Ben should have done much better trying for, expecting, counting on, working toward silver bullets. I sense that Ben doesn't much believe in silver bullets, suspects that any efforts toward silver bullets will be just time and effort wasted. Bluntly, I seriously doubt if Ben knows how to direct a productive, creative, effective little applied research project. Ben reminds me of the remark in the movie about the auto guy Tucker "A well run company doesn't waste time on research". Yes, some research is close to intellectual self-abuse, but it's the job of a good CEO and a good Board to know that this is not always true (e.g., SR-71, 22 nm line widths) and how to make applied research productive, not just throw out the baby and drink the bathwater.
Third, any entrepreneur with a good chance of doing well has to ask himself at least 10^10th times if he wants anyone like Ben on his Board of Directors. Why? Sure: The CEO and his team have seen an area where they might do better. With the CTO, the CEO outlines an applied research project to get the coveted better results. So, for the next Board meeting the CEO includes some foils on the applied research project. Then in the Board meeting Ben wakes up and starts asking questions: How long will the project take? How much will it cost? How good will the results be? What will be the milestones during the project? What is the project schedule, step by step? Or if an experienced auto mechanic can do a brake job on a Ford F150 truck in the time in the book, why not this 'research' project?
So, the CEO has to tell Ben and the Board that for all of the questions, at present he is comfortable with the situation, will continue to review the situation during the work, but otherwise really has no answers at all for the more specific questions. Presto: Ben gets torqued, brings out his fiduciary powers, fires the CEO, before his stock is vested, and 'gets the company back on a business like, bottom line basis' and just blows it.
Even worse, maybe the applied research has already been successful, and at the Board meeting the CEO introduces the CTO who explains the work and, then, the next step, converting the applied research to production software. Again, just over the software work, Ben does an upchuck.
I just don't see Ben being able to work effectively with things that are new, powerful, valuable, innovative, etc. NSF, NIH, and DARPA problem sponsors can. Ph.D. supervisors can. Successful Ph.D. students can. Research professors can. But I don't see Ben as able.
As long as you don't consider app.net, Skype, Meteor, MixPanel, lyft, ifttt, FourSquare, FaceBook, github, factual, crowdtilt, cohere, bump, box, or airbnb to be new, valuable, or innovative[0].
The question relevant to the OP is what happens after the company is going and suddenly encounters a product from a competitor five times faster and for free. The present A16Z portfolio does not answer that question.
Moreover, what you are pointing to as 'innovative' companies in their portfolio is not much like the 'innovation' of a "silver bullet". That is, the 'innovation' in your list from their portfolio is, on the face of it, just innovation in the business 'idea' or business model and not core, internal, technical innovation such as needed to make some software five times faster as in the OP. E.g., you mentioned AirBnb. Where is the core, technical silver bullet in that business? Maybe they have some such bullets in their server farm now, but when they got venture capital likely they had recently still been thinking air mattresses. Similarly for Facebook: Now they have one heck of a server farm with, maybe some silver bullets, but early on they were something from Zuck's dorm room typing with no bullets visible, silver or lead.
I've seen some silver bullets and created several in my career and simply don't see anything in Ben's background or his many blog posts that suggests he knows how to invent or direct the invention of silver bullets; indeed, what I've seen is that, from his thinking, in any company where he was on the Board, likely he would ruin any efforts he saw to create silver bullets. Ben's technical background is just routine; otherwise, he was a 'business manager' who pushed others to work with "lead bullets".
Learn a lesson: (A) No way, not a chance, will any usual, US, red-blooded, no-nonsense, bottom line oriented business manager want anything to do with any subordinate doing anything the manger would have to think about longer than 10 seconds to understand. They still want it like the Ford factory 100 years ago where the supervisor had all the knowledge and the subordinate was there just to add routine muscle to the knowledge of the supervisor. Such a manager does not want to bet even 1% of his career on any work he does not personally understand. Period. (B) In information technology, venture capital Board members are much like such managers -- they want nothing to do with pursuing silver bullets. (C) Actually, the fraction of the population at all good at creating technical silver bullets is quite small, necessarily so since in this context 'silver' means both valuable and rare. A problem is that if the company, once it's up and going, needs a silver bullet, a Board that doesn't understand how silver bullets are made won't back a silver bullet effort, even if the CEO sees his way clear to success, and will insist on lead bullet approaches.
Again, I just don't see Ben as a silver bullet kind of guy.
Net, the silver bullet business is fairly well understood, but nearly all the venture partners are among the worst at working with that business. Again, good work on silver bullets includes the SR-71, 22 nm line width, and most of the funded projects of NSF, NIH, and DARPA.
In particular, the crucial, core technology of my startup is some silver bullets I created in some original applied math based on some advanced prerequisites in pure and applied math. I've converted that math to software, but to evaluate the real promise of the company just must trace the math. A16H has already written me that there is no way they can consider the original applied math. Okay. But once my company is up and running and needed a silver bullet, and I already know one place it will, if an A16H partner were on my Board then I can see it right in front of me like a freight train 50 feet away at 80 MPH coming right toward me, as soon as I tell the Board about the little project for the next silver bullet, they will in unison upchuck on the Board table and fire me. No way. Not a chance do I want any such person on my Board. I'll let those guys stay busy with another local, mobile, social, photo-sharing app, like Instagram except for video clips or some such.
I'm not used to working with people who are afraid to work with silver bullets. Instead I've been working with silver bullets right along in all my career in applied math and computing for problems in US national security and business, in my Ph.D. work, in likely the world's best computer science lab, etc. Again, from my experience, Ben doesn't look like a silver bullet kind of guy. Neither does A16Z.
For Ben, his background doesn't look very technical, i.e., like someone able to draw from the research library, add some new ideas, and come up with a silver bullet in, what, two months, two weeks, over a weekend, in the shower after a 2 mile run? Next, in the Bayesian sense, there is a high 'prior' probability that a guy with Ben's background would resent technical work under him that he didn't understand.
An entrepreneur needs to know, to be quite sure he has a Board able to work effectively with silver bullets. I know many such people although only a few in business, and all of those people had good advanced technical backgrounds. Net, finding such a person in a VC firm looks tough.
I will add a point from my project: Back there many moons ago, I wanted some info from the Internet and couldn't get it. So, I put my feet up, popped open a cold can of Diet Pepsi, reviewed the problem and some of my best graduate school math, and had some new ideas. For both mathematical reasons and just simple judgment, the technical ideas look at least comparatively good as the crucial, core technology needed for a business to let users get more info from the Internet. Then I noticed that my ideas solved much more in getting info from the Internet than my original problem; such a situation is common because commonly a solution does not use every detail of the original practical problem and, thus, also applies to other problems that share the details that were used. It's like someone cooked up the Pythagorean theorem to help survey farm land in Egypt after the flood waters receded but use nothing about farm land. So, the Pythagorean theorem also works for carpentry, stone cutting, navigation, astronomy, and with some generalizations, much of multivariate statistics, Fourier theory, Hilbert space theory, and, thus, quantum mechanics -- my ideas are not nearly that powerful, just have utility nicely beyond my original question.
So, I wrote the corresponding, core software. Then I needed to put a Web site in front of my core software. Then, given my thin checkbook, reality came between me and my keyboard and typing, and I got delayed. In addition, I ran into computer security problems with Windows XP SP3, some Windows system administration problems with SQL Server, etc. -- more delays.
So, I'm a pre-revenue project that needs just a few more Web pages, one moderately involved and the rest dirt simple, some initial data, and some routine things like a domain name to go live. For where my techniques for getting info from the Internet work, if Internet users, in the US and around the world, like my work, judging from the latest version of Mary Meeker's (KPCB) report, then the project will be able to grow to be a big thing. Right: For a large fraction of what's on the Internet, a much better way to get that info for the US and around the world, 2+ billion people, so, sure, $400 billion. My project didn't start out with any such ambitions; I just wanted something fairly routine, but the usual Web sites were not much help.
It does look like even with my thin checkbook I can (continue to) bootstrap this thing, still, if only for a source of what aviation calls "reserve fuel", I contact venture firms.
I've learned that apparently most of the firms have agreements with their limited partners that they will fund only projects with good 'traction', that is, usage and/or revenue significantly high and growing rapidly. And, then, the venture firms will 'play with the Web site', look at the user interface and user experience, and try to estimate how 2+ billion Internet users, desktops to smart phones, would like the site. Okay. Fine.
But the key to the work was some silver bullets. That is, the market need is easy enough to see; the issue is how the heck to get data and write software to manipulate that data and get the results for the users. From efforts from others, clearly cooking up how to do that ain't easy. For that, my answer is some original applied math -- then the core software just implements the math.
Well, not a single venture firm has shown any interest in the crucial, core silver bullets. A16Z essentially deliberately laughed at any possible role for applied math (NSF and DARPA don't). Okay.
Fine with me to continue on with my thin checkbook, go live, and get some traction. Shouldn't be much more work for me assuming that people like the site at all (even if my math works as well in practice as I hope, still people might not like my site).
But, then, I come to the idea of what might come after a venture check? Okay, there will be a Board. If A16Z were to write the check, then Ben or one of his colleagues would be on the Board. Setting aside A16Z, there would be one or more venture partners on the Board.
Then once the initial crunch after going live dies down, and if the usage is growing quickly I've hired a COO to help address that, it will be time for another silver bullet, in this case, for some quite different approaches to ad targeting. I want especially effective ad targeting but only from my own data and with next to no concerns about invasion of user privacy. I have some ideas how to proceed, but they are silver bullet ideas. If I put up the applied math in a Board meeting, could count with shoes on all the venture partners in the US with educational backgrounds to understand.
So I will be asking my Board to back a silver bullet project for ad targeting. And there's a lead bullet approach -- mostly just use ad networks with their pros and cons. And I can see Congress getting all concerned about privacy and severely constraining what ad networks can do. Tilt; my revenue could be at risk.
Sure, some VCs emphasize how much freedom they give their CEOs. Fred Wilson and Brad Feld do. But a lot of VCs also emphasize how they are 'hands on'. Either way, the Board can fire the CEO, before the stock is vested, and that's me. So, net, I want a Board that can work effectively with projects to generate silver bullets (or to use Mother Goose, golden eggs). For more in Mother Goose, the story of Little Red Hen is too close for comfort.
So, how do I know if a VC can work with silver bullets? Sure, contact them and see what questions they ask and how they react to my answers. Do they ask good questions? Or if they can't evaluate the silver bullets I've already got in production quality software, what the heck are they going to do with a silver bullet ad targeting project that starts from much less? More generally, what would they do with other silver bullet projects?
The US DoD got the scram jet working, after only a few failures. Good for the US DoD. They've been nicely successful with silver bullet projects since early in WWII -- drag free satellites, GPS, etc. So, it is possible to work effectively with silver bullet projects. There are lots of people who know how, and I've worked with some of them in my career in DoD work, in grad school, and in computer science research. And if I have to have a Board, then I want such people on my Board. When I look at VCs, I don't see that they have such backgrounds. I'm concerned.
I read the original article pretty differently. As far as I can see, these all are exactly the kind of lead bullets the article meant. No miracles, nothing that doesn't directly involve making your product better, just some good old-fashioned programming work. The author's point was that if you have an inferior product, you roll up your sleeves and make the product better.
Many times people are all over the place trying to over-pivot, when what they really need to do is focus at the job at hand. If execution is lacking, a new strategy won't help.
At a prior firm, a few of the sales people would come up with new flavors of the month. "We need to Open Source. That's why we lost." "We need to be SaaS. That's why we lost." "Not enough functionality." "Too expensive." The reality is there were buyers in our market, we just weren't winning.
This isn't to say there is never a time for a new strategy - pivoting is vital. It's just to do it at the right time, and have the guts to stick with execution when it's needed. Of course knowing when that right time is doesn't come easy.
On "guts", for a lot of entrepreneurs, they know as they start that if they fail it will cost them their savings, house, car, credit rating, likely damage their career, and maybe hurt their marriage and health. So such a person can't be short on "guts". And unexpected disasters, such as Microsoft's five times faster IIS, can pop up needing guts.
Still, one major source of needing guts is vast projects with half-vast planning. Or just poor projects. Hopefully with good projects with good planning, with some silver bullets in the crucial, core technology, the need for guts and "to stick with execution when it's needed" will be much lower than usually assumed. On "guts", entrepreneurs short on guts would not even have started.
What are some examples of startups doing it (or not doing it)?
For a lean start-up, building the consumer side makes the case for the business side.