“Scientific Method for Startups”, and AMA with Michael Seibel
blog.ycombinator.com
blog.ycombinator.com
The Tim Robinson example in the article recounting the removing of links on the checkout flow illustrates this. He observed something interesting in the data (people clicking on links), but he then skipped straight to a test which removed links and found conversion went up. It's implied that people were being distracted from the main task by the links.
But it might equally be that they wanted to click the links to find some missing information about the product. Or even some other reason. Without any qualitative analysis to establish a reason why people were clicking on them, you don't know if he might have raised conversion even higher had he kept the links and addressed a need people had. This is particularly important in a checkout flow where the numbers of people are comparatively low and behavioural elasticity is also comparatively low.
So I'm reading between the lines here that because qualitative analysis is hard, people don't do it. Maybe that's fair enough, but that doesn't prevent you from kicking the tyres on hypotheses. Quant analysis and A/B testing will only tell you what happened, not why did.
1. Studying other products and reasoning about them.
2. Conceptually dissecting multiple existing products and looking at different combinations of some subset of their components.
Any thoughts?
PS: Loved your essay
Just curious growth wise, did Socialcam have a lot of word of mouth, or was some growth hacking involved as well. 16 million downloads in 3 months is incredible.
Sometimes starting the other way around might be helpful - not seeking inspiration but defining the essence of your value proposition and starting from there.
We often tend to develop horseless carriages.
Generally, I'm not a fan of "scientific method" as a phrase. It's too often used by people dismissing intuition in a very non-scientific way.
On the other hand, I don't necessarily know what this technology is going to mean to end users. They're going to find problems to solve that I didn't even imagine. As we're closing in on a beta, I'm doing demo/interviews as often as I can (a couple of times a week at this point), and seeing where their minds run. As William Gibson said, the street finds its own uses for things.
Once we get the product in the hands of real customers, it'll be far easier to start taking measurements and figuring out what to work on from there. But for now, intuition is the only tool I have.
My question: how do you apply that when you have tens of visitors/users? Not nearly enough to have a reliable ab test.
Edit: so far he answered only one question. Not much of a AMA, right? Maybe remove it from the title.
Quantitative methods aren't reliable with such a small dataset so you will have to rely on assumptions and guesses. Unfortunately this makes your observations prone to bias from your hypothesis, so it's especially important to be mindful of this and keep them separate.
Edit: Didn't realize this was an AMA but hopefully my answer contributes to the discussion.
I think your approach is right. One thing that helps me is to use hypothesis from users/visitors/customer as much as possible. Talk to customers and try to distill what are their hypothesis about a specific problem. Talking to customer goes a long way.
On the topic, I think that famous quote atributed to Ford "If I asked my customer what they wanted, I would be trying to create faster horses". I think is BS (and probably bogus). That ad absurdum scenario would only happens if he asked the wrong questions and/or followed the most superficial answers.
About the quote, it kinds of depends on what do you mean by "want", but I understand and agree of your approach (while still disagreeing with the quote if that is possible).
BTW the quote is probably bogus, as the first Ford cars were slower than horses. 45km/h vs 88km/h (world record)
https://en.wikipedia.org/wiki/Ford_Model_A_(1903%E2%80%9304) https://en.wikipedia.org/wiki/Horse_gait
https://hbr.org/2011/08/henry-ford-never-said-the-fast
https://www.quora.com/Did-Henry-Ford-actually-say-%E2%80%9CI...
I also agree that A/B testing is super important! We are following exactly what you have described here http://www.michaelseibel.com/blog/product-development-cycle-... but for a game, and the results are pretty interesting! Specially because we don't need a GDD (a document describing the game), all members of the team are involved in the creation process (and they feel more motivated to work - and believe me, this is big problem in indie studios) and is easier to propose changes and improvements to the game, since them just need to bring us closer to our goal with the game (for example: 20k ad visualizations / day). I'm anxious to get into this A/B testing phase! We haven't launched yet.
Thank you so much for sharing the knowledge! These posts were really valuable and helpful for me! And I think they are complementary!
My Q: When one a/b tests there is generally a winner, but a better option might be possible if the team spends a bit more time brainstorming and hypothesizing. Do you have any thoughts on when to go with the results of the a/b test vs when it's worth going back to the drawing board to try to brainstorm a better idea?
What people often don't realize is the amount of inspiration you get by discovering your hypothesis was wrong.
How much racism have you faced in Silicon Valley? Do you ever face racism from companies going through the YC program?
Why do some projects evolve into behemoths, and when do you advise people building projects to start talking to users?
This is one of the key take aways from customer discovery - is your customer actively looking for a solution to the pain / problem you've identified. If your customer is actively looking for a solution than it is easy(ier) to identify and talk with them. If no one is looking for a solution then it may be that you haven't identified a real pain, you haven't identified the correct customer, or you are to early.
It's easy to say that you can't talk to customers because the industry hasn't been identified, but that is a bad statement and should be avoided.
Otherwise you still want to follow the general pattern of talk to whomever stands to gain the most from your service. Note that the person that stands to gain the most might not be who you first assume it is.
Going into the market and talking to your customers early on so you can answer those two questions will really help tell you if you have a viable idea.
To better understand how to conduct the discovery and what questions to ask you should read a bit of Rob Fitzpatrick's The Mom Test.
It's hard to give one piece of advice about how you should do customer discovery (without having a high quality conversation about what you are doing) b/c every venture is a different in what needs to be tested. Software ventures tend to need product customer discovery (what you should be building), while STEM companies[1] tend to need go-to-market strategy customer discovery.
[1] An example would be a company that is making an orally delivered anti-coagulant drug. They have HIGH technology constraints that aren't easily adjusted, so their customer discovery tends to revolve around questions about what their real value proposition is (is it pain free needle free drug delivery or no medical training drug delivery or something else) and other such questions.
Was there ever a point when you had too many cooks in the kitchen?