What We Learnt From a Failed MVP | Hacker News Reader

What We Learnt From a Failed MVP

arg0s.in

55 pointsskmurphy22 comments

While lean is generally a great step forward implementation can be quite tricky.

Saw a video last night by Jeff Lawson of Twilio about a lean test he conducted on an email sending API. It's starts at the 9:30 mark:

http://vimeo.com/16306094

He did a very thorough test using Google to drive traffic to his test website along with various A/B tests on pricing.

His conclusion was there wasn't a viable market. Tell that to SendGrid and their various competitors ;<). I've done the very same thing with several sites and was always left wondering was it the test or my idea?

My conclusion is that to do lean successfully you also need to get off the web and get out and talk with potential customers face to face. Sometimes if you listen carefully they will hand you an even better idea.

Yes! That's part of why "get out of the building" is such a popular Lean Startup mantra. More generally, I tell novices to cycle between qualitative and quantitative learning.

For example, I recently had an idea, so I did a bunch of interviews with potential users. That gave me enough information to form a quantitative hypothesis: if I offer users app X, then Y% will use it regularly over a two week period.

Whether that fails or succeeds, my next step will be to talk with users. If it succeeds, the interviews will tell me where to take the product next. If it fails, interviews will help me distinguish between a fatal failure (e.g., nobody will ever use this) or a fixable one (e.g., minor UI issues make friction too high for frequent repeat usage).

>My conclusion is that to do lean successfully you also need to get off the web and get out and talk with potential customers face to face. Sometimes if you listen carefully they will hand you an even better idea.

Can't repeat this enough. TALK to your (potential) customers. There's just too much to learn from them to ignore this step. Not knocking on the mostly valid assumptions one can make off a/b experiments, but to tweak your product to its best potential, you really should hear and see your customers react, interact, and critique your product.

Agree completely. We did run a number of phone and face to face interviews, but in hindsight I feel we didnt reach out to the right demographic. This is something we could have done more of. Thanks for the feedback.

This is an excellent lessons learned write-up in a framework that is very clear. He outlines the initial vision, key assumptions (labelled "big IFs) and then a sequence of five experiments with a clear definition of Hypothesis, Audience, Acquisition, Validation, Execution, Result, Learning. The author is candid and offers a conclusion that I find very insightful:

   "However, one thing I noticed after I moved back to India from the 
   Valley was that we often hesitate to talk about failures, and 
   what we learnt from them. People are much more interested in 
   learning from a success story. However, if you ask me, I’m more 
   interested in the 51 attempts of Rovio than just the one that made it 
   big. It’s the failure to learn that’s a failure."

Thanks for sharing this on HN, I was surprised when I suddenly saw a spike in traffic and tweets. ;)

I certainly think the Valley is more open to discussing failures and what worked and what didn't. In my experience, it's been hard to have honest, clear discussions in India with other entrepreneurs about challenges being faced and what people are doing to address them. There's just a lot of noise in the startup ecosystem. That said, all this is maturing, and I'm sure it'll get better soon.

Well there are people who write about failures and I love to read about it.

check this http://www.cbinsights.com/blog/startup-failure-post-mortem

not to mention this list too http://www.cbinsights.com/blog/biggest-startup-failures

I don't think this had to be a failure. A huge part of the job of building a success is being willing to change what you're doing based on the results that you're getting.

The one thing that I think is missing from "what we could have done better" is this:

Be willing to CHANGE your ideas, big time.

You had people signing up. They wanted something. Instead of sticking with your first idea (video), why not try a bunch of stuff until something worked.

You have to be creative, flexible, and determined in order to make this type of business work. You had SUCCESS with many of the tests -- don't throw those successes away just because one test failed.

People didn't want video. Give them something else. Does it really matter what it is? Does it even have to be charades? Be willing to CHANGE on the spot in a BIG way.

Think about this:

1. People sign up and say "yes, I want to play charades".

2. They land on a page tht asks them to play charades. How the heck can you immediately engage them? By asking them to turn on video? Or...???

If you're giving up because people don't want to turn on video -- then you may as well not try. Think about this: If I want to have "online boxing", should I give up if people don't literally punch themselves in the face, in order to simulate real punches? Sure, it sounds silly, but it's not that different than asking people to turn on video. Online video can literally be harmful to people, especially when it involves strangers, and even when it involves friends and acquaintances. People don't want video charades. That doesn't mean they don't want charades.

Don't get stuck on the wrong thing!

You said exactly what I wanted to say...

An alternative improvement that I would feel comfortable -

1) Instead of email id, just ask "enter a nickname". More people would feel comfortable this way.

2) Once I'm in - "Welcome to your private charade room. Invite your friends to join your private charade room by sending this url".

Provide a private url to that room that they can send to their friends. Let them send it by mail. And don't ask me to enter my friend's email id or link to my facebook / twitter accounts. Many have already fallen for this trick from Linkedin etc and paid heavily for giving them access to the friend's ids.

Now make it easy for me also to enter this room using the url. If my friends enter the room, we are ready to play.

Allow me to create multiple rooms. So I could play with my family, my friends or colleagues at different time. And all through the day I could watch which room has member and start playing ...

3) And now provide an option for them to join a public room... Here mention clearly that your video is now visible to the worlds and could even be recorded ....

This lesson shows the difficulty in testing of any game or pop song or any other product that is really "fashion." They do not follow the same rules as a typical business model would. You're not solving a problem or getting a job done. You're providing entertainment. You have to try it and see if it works. You need the vision of an artist and quite a bit of luck.

Nicely explained.

One thing I have to tell novice experimenters a lot: A failed experiment is not when you get a result different than the one you wanted. A failed experiment is one where you don't learn what you set out to learn. Which can also be fine as long as you learn how your next experiment can be better.

This was a successful experiment.

Agree completely. I left the misleading title in there since not everyone is familiar yet with some of the lean terminology.

I did include a para further down in the article explaining this further. It's the failure to learn that is failure.

http://arg0s.in/what-we-learnt-from-a-failed-mvp.html#was-th...

"Failed MVP" is a contradiction in terms. (The V.)

Sustainability implies pivoting into something people will pay for, you can build and convince them to use before cash runs out. If the team is highly motivated, they will not disband on a single failure but will find something else that people want.

I find it difficult to come up with the right validation criteria. Would you share your thought process? How did you come up with specific numbers?

  Of a sampling set of people searching for Charades online, at least 25% will sign up
  to check out the game.

It's fairly subjective at this stage, and is more of a judgment call than a science. I went with the following assumptions while putting down the 25% metric: * Conversion rates on vanilla sign up pages (with not a lot on the page) are generally not that high. I'd read that 8-10% was common. * Users already searching for charades online would be motivated to check out the ad (in my mind, this justified looking at a higher rate) * Since this was SCM based, it was not granular from a demographic perspective - likely had a mix of early adopters, mainstream users, etc.

Hope this helps. I'd love to hear how others are doing this.

This is definitely a good exercise to do.

I just went through a similar process and did a write-up of my own - https://medium.com/how-to-succeed/86e70e2c33c1

I like the writeup, but one question I would've liked answered is: does the OP enjoy playing charades?

To me, it seems like the best products are the ones that served the creators' wants and needs. Github was created because the founders wanted to share code more easily. Facebook was created as a better way to (eventually) get laid as a college student.

If you really love playing charades, and more importantly, love playing charades so much that you'll do it online, even with strangers, then the path to a viable MVP seems clear. As it currently reads, the OP's analysis seems sound, but without much of the passion that seems needed to drive a product.

Passionately loving the domain is a double-edged sword. Sure, it can drive you to make something awesome. But it can also drive you to make something awesome for you personally, but that nobody else gives a shit about. Or, more subtly problematic, it can lead you to make the software equivalent of quirky indie movies that are, at best, marginally profitable.

I also think passion is something you can develop over time. I've worked on a number of things where I had no initial interest. But I do like making users happy, so I get excited about the domain via seeing what gets users excited.

LOL, yes I agree that my writing was a bit dry. But we all play charades, and are passionate about it. That said, I think there are enough examples as well of startups that have succeeded where the founders themselves were not part of the primary target user community, but they developed a deep enough understanding of the users needs to build a successful product.

Incidentally, I OH this on Facebook: We tend to underestimate long term impact of reason and short term impact of passion.

In some sense it is almost better if the founder is not too attached to the idea. I don't see how one user (the founder) being super excited translates to a viable MVP - but I could see how it could lead them to ignore the results of their experiments and the actual data.

"I don't see how one user (the founder) being super excited translates to a viable MVP "

Less excited people simply won't bother building it.

It's that simple.

Less excited people won't know what makes some problems acute and some not so much.

Reply on news.ycombinator.com