Hacker News Reader

Comment by halfmatthalfcat | Hacker News Reader

I’ve heard this a couple times but it still makes no sense to me. How do you test your product hypothesis without a product? You’re essentially testing the viability of an idea with what you’re proposing, which is vastly different than having an actual product with features that you can iterate on when given feedback.

I think for very niche categories, a landing page might be the only thing you need but I honestly believe that if you’re pitching a product with features, people want to see that product, not a page with text.

View on HN

It's a false dichotomy. In the real world you can't fully test your product hypothesis before building anything, but you also can't build a complete product before testing the market.

Instead, it needs to be an iterative process. Build a little bit of product, tease it to the market, gauge interest, engage with customers, adjust trajectory as necessary, and continue to iterate.

Testing your business hypthesis before building anything is prone to generating false signal. Some times your customers don't realize they need what you're building until they see it and use it. Many times your customers will praise an idea, right up until you ask them to use it or pay for it.

A good example might be the recruiting company started by a famous HN commenter around 2015 that tried to use programming games as a recruiting filter. The idea received huge amounts of praise on HN and large numbers of waiting list signups before they released anything. The demand and hypothesis appeared to be validated. Then they switched to the other extreme, building enormously complex systems for years without actually selling anything to their customers. Eventually they shut down because they realized the market didn't actually want what they were building. The initial signal was misleading, but going heads-down to build a product according to the initial signal was also misleading. A better approach would have been to start recruiting up front and slowly iterate on improving it with programming games, rather than going off in the weeds to build a product that no one actually wanted.

This is not true. There are many many mockup tools that you can use to test without building anything. It's a good way to separate the technology decisions from the business decisions.

Mockups are part of the iterative process. Building a mockup to test is part of what I was describing.

The point is that you can't simply make a mockup, assume it proves your hypothesis, and then go heads-down building the exact product shown in the mockups for 6-24 months and expect great results. Instead, you need to iterate both the mockups and the product in parallel, engaging with the market along the way.

As in the example I provided, sometimes customers will claim to love your mockups but then decline to pay for the product when it arrives. It's one of the first things every product manager learns in the real world.

> It's a good way to separate the technology decisions from the business decisions.

Trying to separate technology from the business decisions is a mistake. At the start of the process, you need technology, product, sales, and business to be tightly intertwined.

Dividing the labor and sending different roles in different directions is a mistake at an early stage company. You need to get everyone working together to iterate over and over again.

I don't mean a hard separation in terms of roles, but I agree with your general sentiment. The reality, in my experience with startups, is that an MVP is built by an engineer or a couple of engineers and it's launched as quickly as possible to get market validation. The thought is usually along the lines of "we'll launch it, get feedback from customers, and then quickly iterate." The reality is usually that the MVP is launched and the company is stuck supporting that MVP and the company is afraid to make changes because they got a customer to give them money (enough perceived validation) and they're stuck supporting the MVP they've just launched instead of iterating. I've seen it so many times that I refer to it as "the MVP trap".

Also, User research can be done in a way to account for the fact that users will say they love something, but actually never use it. I've done it many times.

All of these things that you've mentioned should be done in tandem, but in practice, that's pretty rare. The good thing is that it creates market opportunities for other companies. Amazon is a more extreme example, but it's basically the business model of AWS. Another example is Terraform. HashiCorp has done a poor job of keeping up with user needs so there are add-ons or straight up replacements coming out of the woodwork to fill the feature and experience gaps that have been around in TF for years and years. You will see more competitors to TF and other HC products coming up for exactly this reason. They aren't even the close to the most glaring example, but just the first thing that came to mind.

> Some times your customers don't realize they need what you're building until they see it and use it.

This.

That's one big part of testing the hypothesis that I feel a lot of people (including mentoring organizations) really don't get.

In an ideal scenario you've clearly identified a problem and can accurately gauge the validity of your solution by just talking about it with prospects, as they can easily relate. I'll label this approach the "Let's-imagine" validation.

In other situations though, target customers aren't even aware that they have a problem (maybe the problem is not immediately apparent) and it can be difficult for them to understand your value proposition, since they've "very successfully been using Excel for that for the past 20 years". You still need to somehow relay that there's a reality where things are significantly better than in the status quo. In some such circles, relying on simple conversations or mock-ups can at best prove very difficult (people won't give you time to wander in fuzzy hypotheticals), or at worst be plain misleading (they will dismiss your idea as irrelevant). In situations where you can't rely on your prospects' imagination, you need to bring proof and the mvp plays a more important role. I'll label this the "Show-me" validation.

"Let's-imagine" and "show-me" are not absolutes. They're on a spectrum, but it seems that the advice to validate by talking to customers is being taken to some dogmatic extremes sometimes. It's something for founders to be aware of when they receive canned advice. Regarding your own solution, you need to be smart to identify where on the above range you fall.

If target customers are not aware of a problem and think excel is the way to go, how, even with a product do you get as far as to show it to them? And then what?

> the recruiting company started by a famous HN commenter around 2015

... started by 2 famous HN commenters. Why did you chose not to name these HN commenters?

I believe you are entirely correct: it is testing an idea rather than testing and iterating on features. That's the point: people who love programming might skip directly to testing features without testing the idea. Testing the idea is essential and can be done 10x faster and cheaper than testing the first feature.

Okay, let's take Roblox as an example since they're about to IPO. They've clearly made an incredible product and built a developer ecosystem around it.

When they were starting several years ago, how would have tested "the idea"?

That's just survivorship bias though. A lot more ready-coded ideas fail which means a lot more wasted time (if making money is the point). I agree that not all ideas can be tested that way, some ideas just have to be experienced. For the ones that can be tested in their idea phase, it seems like a good way to prevent doing too much when the interest just isn't there. At the same time you'll probably miss out on some ideas that need a bit of a runway first, but you're minimizing for downside risk instead of maximizing for upside potential.

How is my question survivorship bias?

Other than the sense that any question asked is one of a multitude of candidate questions that "survived" the decision process and made it out into the world, I don't see how asking how a company's "idea" may have been tested is survivorship bias.

You are right, that sentence and the one after that are irrelevant. My mistake!

No worries!

They could have put out a site with videos of the product and experience akin to what Nikola did with their trucks.

Vaporware is an extremely common tactic in sales.

Instead of asking, "How do I test Roblox?", I'd ask:

"Is my idea testable with the time/runway I have?"

Assuming the context of a solo dev that just wants to make money: info products, simple apis, or simple apps should be easier to test.

I think that doing lots of small projects and launching them is probably better than working on "the one thing" - until one of them becomes "the one thing". This gives you more opportunity to test ideas and test marketing (which is probably more important than your idea). The canonical example of this is probably Pieter Levels [1].

Let's call this the "Test with Teeny MVP" [2] method as a opposed to the "Test with Landing Page" method. Important note - I suspect that testing with a teeny MVP is easier than a landing page when you don't have an audience. Interesting tweet on that from Rob Walling [3]:

"Out of nearly 1600 applicants to @tinyseedfund we chose 23 exceptional companies to fund. Of those 23, one (maybe two if you stretch) built an audience before launching their SaaS.

Audience helps, but so much less in SaaS. I've been beating this drum since 2012."

[1]: https://levels.io/12-startups-12-months/

[2]: I'm sure that someone will point out that "Teeny" is redundant here. :)

[3]: https://twitter.com/robwalling/status/1306591312498405376

> Instead of asking, "How do I test Roblox?", I'd ask:

> "Is my idea testable with the time/runway I have?"

I'm already ramen profitable with what I'm doing. The vast majority of my ideas are not ideas for new businesses, though a few are.

The idea I'm testing in this thread is, "looking at successful businesses, how would rigid Lean Startup practices have worked in their early days?"

Tinyseed isn't terribly interesting to me in this context since they have a very narrow SaaS focus, none of them have had an IPO and Rob has a large podcast he uses to promote the companies. It's a great strategy on his part, but it makes it much harder to figure to know how much the success of companies they invest in is due to their ideas.

Something similar could be said of Jason Calicanis's launch.co, though I bet it will have some massive wins in the next few years.

Roblox's history actually goes back decades. In 1989 Dave Baszucki cofounded Knowledge Revolution and created Interactive Physics, an educational physics simulator. Seeing how students used the software inspired him to cofound Roblox in 2004.

Lots of market research, surveys, and analysis of market trends. And then you build and hope that analysis translates to customers.

Most people here have the right intention on testing your idea/concept before building, but they forget the world is 100x more competitive than it was 10 years ago. All the low hanging fruit ideas are gone. Yes you can obviously test some ideas easily but most of them were executed already.

To succeed now, you need to tackle the “high hanging” fruit ideas/problems and a lot of those simply involve more complexity/building and are not as easily testable via a prototype. Instead you need to invest more into market research before you build a huge chunk of it (while continually doing more market research along the way)

I mean, you just make up features and ask customers if they want them before writing a line of code. Check out https://buffer.com/resources/idea-to-paying-customers-in-7-w... from buffer.com for an example.

It was literally the landing and pricing pages first, functionality came a few weeks later and only after people were clicking on the (non-functional) pricing plans because they wanted to buy the product.

How do you translate product efficacy by an anonymous person clicking a buy link on a pricing page? All you are doing is gauging this ephemeral, uninformed click which has no direct correlation to what the product will actually _do_.

It seems more of a crap shoot to gauge interest off of random clicks on a buy page versus having an actual product with actual user feedback that is actionable.

Well, obviously. But there is more to it than just those two in isolation. First off, one anonymous person clicking a link is just noise. But if you have hundreds clicking a link per day that gives you a much better indication you might be onto something.

Further, how much time do you spend on making a basic landing and pricing page? Two hours? Four maybe? The original comment was talking about programmers working for 6+ months on a MVP product before showing it to people. You could launch hundreds of landing pages in that time, only limited by how many ideas you have. Even though the chance of a "hit" per landing page is probably lower, the sheer volume can more than make up for it. The landing plus pricing page model also immediately informs you how much people are actually willing to spend on it, which (if higher than zero) is a much higher indicator for product success than "I think this is a super cool feature".

Finally, I notice that you are talking about "product efficacy" while the original comment is interested in making money. The two are related but not the same, since there are tons of crappy products making a boatload of profit (comcast anyone?) and also tons of great products that struggle to make money (see the ongoing troubles of open source developers on this very forum). Either goal is fine, but optimizing for one while thinking you are then also optimizing for the other is a recipe for disappointment.

I can't get over how much of a dick move that sounds haha. Hey you like this product? You want this do you? Sike it doesn't exist give me your email address lol

Buffer was created in a different time/year in history. It was created when social media management was in the infancy and there were very few tools that did what they did (I know because I created a marketing product around the same time as them)

Now there are way less of those ideas available. And the ones that are available can’t be tested with a simple landing page.

At the end of the road, you still have to make a landing page showcasing what your product does, what problem it solves and why people should sign up. Usually that landing page doesn't actually have any features, just shows people what features they will get.

So build the landing page first. Showcase the features you are going to build and see if you can get people to sign up.

This is right up there with the "to gain users, build a product that users like by talking to users" paradox.

It seems strange to me that you would build without already having a problem to solve.

Not GP, but here's my 2c:

> How do you test your product hypothesis without a product?

You can't. Actually... it depends on how you define a product. The way I think of it is: a product is merely a vehicle for the added value. With that understanding it becomes easier to decouple the value from the actual product. Therefore the MVP is no longer a stripped down version of the product but rather the simplest tool you can build to provide [that] value.

I think I read about it in Eric Ries 'Lean Startup' or so. He narrated his experience of building a product, making a signup page for that product. Nobody pressed the sign-up button. In this case the signup page with the button (and then transfering to 'get notified when it is ready') would have been enough.

I did a little focus group work years ago. Companies pay people to listen to a pitch or watch a demo and give them feedback.

Reply on news.ycombinator.com