The danger of the mediocre success when testing startup hypotheses (2023)
pivotal.substack.com
pivotal.substack.com
If you think that scientific rigour will help you avoid the need for good judgement, you're in for a great deal of distress. Edited for typos.
Broadly, assuming more data and tighter calculations == more certainty is a folly endemic to engineering-focused organizations and industries. In reality, you're increasing precision without increasing accuracy: a dangerously misleading state. It's understandable: organizations obviously need to play to their strengths, but there's real danger in succumbing to the "when you're a hammer, everything looks like a nail" mindset. Firstly, It's much easier for non-engineers to parse the difficulty of engineering problems than the other way around. Outsiders easily see that they don't have the requisite chops to do engineering work. Secondly, cutting through ambiguity and unpredictability to reveal factors you can control is critical to engineering, while non-engineering jobs like management, design, and community outreach are hard because you must confront those things. Engineering types often reason about non-engineering problems either like they have the same level of predictability, or simply disregard the ambiguous and unpredictable factors because they don't fit into an equation.
It's easy to see why that micromanaging non-technical manager screwed up by insisted on using some technology or approach they read a snappy article about; an engineer's authority is cut-and-dried in that situation. It's harder to see why purely formulaic approaches often fail dealing with people, nature, markets, etc. Most things in the world are far more complex, temperamental, and less predictable than cache invalidation.
-- N.N. Taleb
The thinking so different than the candy store margin mentality of 3 generations ago.
Love it.
This is what all big company PMs and directors of engineering believe. But their judgement (good or bad), their metrics (scientific or pseudoscientific), their career trajectory (up or out), whether these are known or unknown in the first place, and their actual material success: who knows how correlated it all is. They don’t have the words to express this stuff, so they wouldn’t be able to see the difference between science and pseudoscience for example, and you'd have a hard time communicating this thing about science to these people.
So while you are right, it’s not saying much that the real problem is communications, and that is the mainstream opinion of people who study microeconomics / the structure of firms. Put differently, people have built cathedrals of bullshit in their minds in service of the status quo where they “test” “everything” in lieu of having falsifiable, forward-looking opinions (aka judgement). You can’t just go, Martin Luther all of corporate white collar bureaucracy.
Scientific rigor is inseparable from good judgement.
First, you need a sample size that will produce a statistically significant result. Cold calls are expected to have a low success rate and be highly variable. 2 in 100 could very well be a massive success. 0 in 100 wouldn't rule out its viability. If instead you had a sample size of 2000, then you'd get a very good signal to noise ratio.
Then, you need controls. Look up statistical design of experiments (DOE), you'll find efficient ways for finding how much various different factors affect your results. Basically split up those 2000 calls into groups and vary things as you go, so instead of testing whether a specific cold call technique is working, you can see if cold calls in general work.
Finally, understand what success means. A business does not need to rigorously prove that it's methods are optimal, it needs only to find an adequate strategy to achieve its objectives. You should know what it costs to make 2000 cold calls, you should know how much revenue is generated from a sale, there should be a specific threshold where the number of sales justifies the cold calls. You should be doing a process capability study that you are sufficiently over this threshold that, given expected variation, you will remain above it most of the time. What is your Cpk? How much does it vary among the different sub-experiments you performed?
At the end of the day, you're still going to need to make decisions with incomplete information, but don't pretend that you're making a data driven decision when you're not. The worst thing is not an inconclusive experiment, the worst thing is an experiment with an erroneous result you mistake for conclusive.
In modern statistics, even a small number of samples can be considered enough to get a satisfactorily small error range, as long as the sample is random and representative of the population. I would think 2,000 samples is far more than strictly required if you're able to sample from your target market.
Of course there are the low hanging fruits, which scored the clear success. Most of the tests although were not a clear success, but just a slightly better conversion rate.
Multiple tests applied after each other slowly increased the overall value. If taken the advice not to implement the changes of the small success, the overall amount would not have changed.
After all, pretty much all of writing, music, product design, prototyping, etc. is iterative. It's a purely investor mindset that would have you think that small optimizations do not pay back. Having that mindset in creating something is a great way to give up after one draft of a good idea.
How much are you willing to spend to reduce the downside risk, and how many “good” experiments are you willing to throw away in the process?
What the author means is ‘get real about continuous learning’
Of course it takes money to run experiments you can actually learn from, and the article is bereft of practical advice about doing this on the cheap.
However, I clicked. But you don’t have to.
Are you referring to the slightly edited title someboty has put here at HN? But still, no dissonance with the content
The updated title is much better.
Did you read the article? Running experiments means you need to have a null hypothesis - and in a sales conversion experiment, for example, the null should be calibrated to ensure your conversions are scalable wins and not just one-offs.
Mediocrity is often enough to put food on the table. The world is full of companies aiming to survive for a few more quarters with their little mediocre product. And once a company embraces that idea there's usually no going back.
Like say, having a side project or hobby. If you try something and find you both don't enjoy it and fail miserably at it, that's probably a sign it's not the right thing to be doing. If you try and succeed beyond your wildest dreams, then you'd continue.
But if you're in the middle and become moderately successful, then what (as the article says)? Do you see that as a sign you've made the right decision and should continue? Or a sign that you'd be better of trying something else?
It's all too easy to take it as a sign that "things will work out if I keep going down this path" and plow weeks/months/years into something that may not pay off in the end, or be best suited to your talents and interests.
Startup companies generally shouldn’t do science, but that same interactive process should guide you. Because when you start you never know enough.
I have encountered this "mediocre success" many times in AI solutions due to lack of problem definition. For instance, now with LLMs is very easy to write a prompt that gives you the output you want in 5 or 6 examples you have in mind. The problem is to build up your testing scenario from there, and gather as much data as possible until you make it representative of your use cases.
That is the only way to actually test your prompts, RAG strategies, and so on, instead of buying the last CoT-like prompt trend.
…in a zero interest rate environment where there is a bunch of money sloshing around looking to roll the dice multiple times.
I agree this is the right thing to do, but it isn’t always possible to do for a given experiment. For instance, with the cold-calling example, how do you ensure that it’s either a smashing success or definite failure a-priori? Is there a principled way to choose the threshold number of conversions? Do you just pick an arbitrary number? What if the result is just below your threshold? With some experiments I think the only way to win is not to play. Happy to be corrected if anyone has a solution
The successful experiments I've seen look like this: hire 2 SDRs at the time, focus them exclusively on outbound, then measure their results to identify the CAC. Then, you can make a call if that CAC is something you want to scale.
The failed experiments look like this: the founder doesn't want to invest too heavily in something that might not pay off, so we hire 1 SDR and they have to commit 50% of their time to following up on low quality Marketing leads.
It's not so much about the number itself, but more about the experiment design that will give you clear results
Correction: It's anathema for VC-funded startups whose backers need one or two to succeed to unicorn size to recoup their losses from the other 98 attempts. For self-funded or privately/bank financed startups, it's fine to stay low profile and grow slowly, as long as the investors get their money back and everyone gets paid fairly. You may not end up a billionaire, but hell, even a small trades shop can reasonably be worth a million dollars or two after a few decades.
It's after all how the many, many "hidden champions" of Germany's Mittelstand got to where they are: start small, focus on extremely high quality products/services, and dominate entire markets even though no one but experts of their respective fields have even heard of them. And that model working out isn't restricted to Germany either... Japanese YKK zippers are the best example there [1].
A side note: if you don't have VCs or, worse, the vipers of the public stock market breathing down your neck, you have so much more freedom as a founder. You can do what's best for the company long term.
[1] https://slate.com/business/2012/04/ykk-zippers-why-so-many-d...
Take a company like Pdfsimpli for instance.
Author argues for trying to be objective about the viability of business ventures here. It's easy to tell yourself lies that doing just one thing a little better on the execution front will magically triple your conversion rate.
Silicon Valley normalized the idea that if you follow the scientific process (The Lean Method, finding Product/Market Fit or whatever this garbage is known in the latest terminology), and burn out for 10 years of your life, then you WILL achieve success. Most people are blind to survivorship bias even if they know the phenomenon. On top of it, incompetent, scumbag VCs muddy the waters with their snake oil "wisdom" which bright eyed, impressionable tech bros gulp down with passion.
If you achieve a "mediocre" success, which allows you financial independence, then you HAVE achieved success, regardless of what the VCs are pontificating. They'd scream because their investment didn't 1000x. Fuck them and continue living your life. Don't burn out for these soul-less MBAs.
And say you achieve the outsized "success". What then? If you go with the history of silicon valley then your so-called success is going to be a shit deal for workers, consumers and the environment.
The author is right; you want clear signals, not noisey ones. If you're looking for the light but keep seeing shadows you need to turn around.
The same is true for people who launch on product hunt, and get 50 signups. Success? If you're selling $1K deals, yes. But if it's just for $4, or free... do you have PMF?
but to be fair, if you can get a small group of people engaged, and you can tease out why/how, it usually stands that others will respond too. But not everyone will need it, and those that may need it may not be in market for it.
Sometimes a product team will "buy the business", eg offer credits or funding to a prospect in order to get their logo. This muddies up the entire experiment. Is the product worth X, or not? Does it solve a problem for customers that is worth X, or not?
For b2b sales, selling 1 or 2 can also be exceptionally costly because you are on the hook for long term support: and if you bought the business, you are now underwater from day 1 with a customer who has you over a barrel. Worse, your sales force now think that every deal can be won on cost, and will start to show signs of deal fever.
It was incredibly damaging and would now be a red flag for me when looking at future workplaces.
It was a viscous cycle where the sales team promised all kinds of customisations to land deals. The software team would be lumped with the responsibility of delivering many of these while not getting time to refine the core products.
The software despite being critical for ultimately delivering also was still seen as a "cost centre" because on paper the credits were given away for pennies to land the "juicy" consultancy with it.
If any deal was tight, the sales team would just give away more software credits / promise more software customisation to land it.
The result was the consultant side of the business acted like they were responsible for all the revenue despite the software delivering a huge bulk.
On paper the software didn't bring in much because we had a sales team effectively giving away our software.
I've written previously about the dangers of courting a tiny fish from a big pond which is related. The one-sided nature of the emerging relationship where to the small supplier the contract means everything, but to the mega-corp it's actually just "that thing that Bob from Sub-division 3 uses" and doesn't actually have mind-share, because very little software is actually ever rolled out "org-wide". Mega-corps rarely behave in such a synchronised manner and chances are you're not really selling to "MegaCorp", you're just selling to Bob. And when he moves on, there's a good chance that no-one else has even heard of you. If you're lucky your invoices will still get paid for a while after despite no-one now even using your system, if unlucky you'll have to fight twice as hard now to get paid for the outstanding invoices, despite the undercut price you decided to charge Bob because you thought you were getting your foot in the door at MegaCorp.
That seems like a failure to define success in the first place. That seems like doing an experiment and then deciding how it made you feel. You probably want to declare beforehand what your minimum number of sales would be in order to qualify as a success. I agree that the results can still be ambiguous, but the situation he's describing sounds like a poorly designed experiment anyway.
It is hard to define that and sometimes can be noise or mask important information.
For example, some products have long conversion times due to pricing. This is a critical metric to follow, but would be ignored by the metric you defined.
You are in precise agreement with the article's own text: "So when a startup comes to me with an idea for an experiment, the one thing I tell them is: make sure that there’s a well-defined distinction between success and failure."
I mean, my corporate job is realistically pretty cushy and as the sole income earner, my family depends on me. I'm sure I could do well on my own, but it's hard to predict the success of any venture. Failure I actually don't mind, because the answer is easy... just get another job. The worst possible outcome is 'success' where you have a customer base and demand, but it's just never going to be enough to make up for that lost income. That would truly be the worst. You can't just quit without damaging your reputation; on the other hand, selling would likely be difficult without significant recurring income.
There's a book for that https://www.themessymiddle.com
what you need is conviction. if it works for 1/100 calls, and you believe in the idea continue iterating. Also, observe what your peers are doing, you probably don't need/want to innovate on sales or business model.
The danger of mediocre success that the author speaks of is actually the danger of indeterminate market movement being identified as meaningful in a naturally open system, then attempting to draw meaningful conclusions about the data when the most likely cause for, "mediocre success" is you simply haven't had enough time in the market. Many of the traits we associate with good business require word of mouth and first hand or second hand experiences with a company's output/services. Accessibility and availability to your client or consumer is the primary reason businesses fail, and on top of that, most marketing tactics suck valuable equity from the businesses, to the point where people are hired on to determine how much money it will take to gather more clients to cover the costs of the campaign.
This is nothing new.
Frankly, I have found networking events and bars to be the best generators of clients. I never openly advertise what I do, (except technically in this comment), but often times I instead talk with people and ask what they do for a living, if they enjoy it, etc. I could make the argument that I waste my time at these places, but ultimately the whole reason I even own the company is from being at those places and chatting people up, taking them out of the crazy warped reality we get piped in from mass media.
This all said, the article just goes to show how many corporations actually do more harm than good to themselves based on the data they collect and what they believe it correlates to or what the causation may be. These deterministic representations of what reality might actually be for a business owner, or anyone in any situation, is a neo-classical example of Uncertainty Principle imo.
To loop back to the top,
My concern is the author suggests that a business model is created and built when you don't have clients or a working idea of how to directly target them in the first place besides a phone call. The best advertising is client word of mouth,and time in the market. You won't have either of those things until you focus on making one client at a time happy, disclosing what you can and cannot do, laying out realistic expectations and deadlines for them or you, having a contract for any services rendered, if they choose a competitor after signing for the same purpose, that sort of thing, and perhaps most importantly having a business centric understanding of if it makes sense to have the upkeep costs of an entire business as opposed to self realizing what those costs can be up front, clausing your contract as a contractor to cover those costs you might write off in the process of the project for the client, and sticking it out as a lone wolf who gets the job done properly, and as defined by the contract ,when needed.
Maybe my comments will stop someone from setting up a LLC or S corp. To be honest, I wish I had never done so and I should have just ate the contractor taxes instead. Currently, there's a emphasis on volume of production with these corporations, so the cost of paying 10 engineers for a year is much less the cost mentally of trying to do all these things yourself, one client at a time, as a business that has all the same tax benefits or penalties of a larger company with definable cost of goods sold because those people go home at night instead of you the business owner who might pull all nighters or get anxiety over things you can't get finished due to unforeseen circumstances, and no team to back you up.
In the end, the best way to start a business is to gamble a million Dollars with a need in the market for what you offer, at a price clients are willing to pay, and a level of attention to detail they might not get from a larger company in exchange for building trust with you as opposed to a market leader. You hire a few really good, overqualified engineers and people who are willing to tell you then you are doing a bad job or wrong, and you take their collective advice. Most of the time, you are going to be the one who needs to improve, and most of the time, you aren't going to like that feedback as a business owner. It will confuse you, you will want to give up. You will have sleepless nights, you will have nightmares about the most superficial things such that it really wouldn't be much different than getting emotionally invested in Dance Moms even though you've seen all the seasons by proxy 4 times already.
But, those people have ideas. Those people have visions, and ultimately, the best way to run a business is to have enough money to accept the business might not pull a profit in year one if you're going to be nice to your employees.
So to all this I suggest:
Don't start a business unless you really, really have a excellent focus on what you want to do with that financial vehicle because depending on how much money it is we live in an an economy where capital gains are taxed less and require less actual work than actually breaking a sweat or going to a job, assuming you have enough money to invest in the market.
I'll probably get down voted to hell for that last quip, but when my investment account had larger gains in a month than I made in a month as a business owner, off of only a 10,000 investment, I felt pretty stupid. Obviously I can't guarantee that profit/loss, but that's why you make that money in the market while you can, quit your day job if the market investment pays more, and go find something amazing and new to build that just might change the world.
I truly apologize for this.... Block of text. This article just seemed to skip over some really key things that are a reality for many a small business owner.
My experience with it was even after charging $200/hr for data collection, for the better part of two years, having a single client was all I needed. Looking back, I would have happily taken a larger tax hit as a contractor than go through the trouble of creating an entire company. I reinvested all that money into development kits, educational resources for myself, and now I'm sitting at home without a job this year, coding and learning whatever I want to code in the hopes that I develop something people appreciate and enjoy using, or finds some wonderful niche purpose worth charging money over.
If I cared about money, my business would still have clients but I'd be much less happier and I'd have the same number of employees -- myself.
Owning a business to me means putting out quality products that do what people ask, runs efficiently, is easily read and understood, and meets customer expectations in a timely and financially sound manner.
Hiring another employee isn't gonna do that unless I can afford good employees, and I can't feel good about what I pay good employees unless I have enough money to pay them what I would like to pay myself. The expectation being that they take the same level of time and care that I would if their task was my own, so I don't have to go review all their work all the time.
I'd be happy paying devs 400k/1m/year+ and having the best team of devs in the world and only taking home 100k as their manager, if it meant we make quality products, and everyone who works there is happy, and some of the stress of being a business owner could be relieved.
Sure I could take more money home but ultimately my goal is to make sure we can do what clients need us to do and set realistic expectations and at the end of the day that's a high price to pay upfront for services like a company like mine would be, but cheaper than a lifetime of dealing with errors, code problems, you all know what I'm talking about.
Have a good day folks and don't be afraid to charge what you are worth for your services. Yes, there are people out there developing products making ridiculous amounts of money, money you could stockpile and turn into your own business someday without worrying about the struggle bus mediocre success, limited working capital concept this author talks about.
> That’s the danger of the mediocre success. The point of startup experimentation isn't the success itself; it’s the learning that comes with clear-cut success or failure. You don't really care about the sales revenue generated by your first two reps; you care about whether this is a strategy you can scale to dozens and then hundreds of reps, or whether you need to use a completely different strategy. It’s all about the learning. And mediocre successes don’t give you any learning.
Eh, that is doing OK to me, which is better than some nebulous "mediocre success". If you are at least saving for retirement, and able to support a family, that's actual success in my book.
The typical startup outcome, where you try for a couple years, it fails, and then you move on to a corporate job is not terrible.
What's really bad is when you make enough to just barely get by, paying yourself a sub-market-rate salary, and keep limping along for a long while. Waiting for that "success", that big breakthrough, "just around the corner". You can be stuck doing this for years, out of a misguided sense of loyalty, determination or whatever. And then you are not saving for retirement, and letting the best family-creation years (if that is a life goal) pass you by.
That, to me is "mediocre success".
Ask me how I know. On second though, no, don't ask me how I know.