Comment by kareemm | Hacker News Reader

Comment by kareemm | Hacker News Reader

Great read and bang-on (I'm running product on the fourth software business I've started - the first three were sold). But there's a missing piece to the puzzle though. The author talks about using data so that you don't fall victim to bias, but he doesn't talk about how to do that.

In my experience to ensure data-driven product decisions you need both a system and process to:

1. Collect feedback

2. Organize it to be useful

3. Analyze and prioritize customer problems

4. Use your feedback to validate problems and solutions directly with customers

5. Communicate status to your team

6. Close the loop with customers when you build their requests

At the early stages you can do this with a spreadsheet or Trello board. Eventually you'll graduate to a tool or in-house solution.

Here's a piece that dives into the details about how to build your own system to collect and organize feedback so you can eliminate bias and drive product decisions based on data:

https://www.savio.io/blog/product-leaders-guide-customer-fee...

The real question is how do you get data that customers are spending more time in your product because they are annoyed as hell

Your A/B test just says “look, they spent more time, do it again, they love that antipattern!”

That’s what qualitative data is for. At the very least, talking to customers and reading their feedback. Preferably followed by organizing that data somehow, and using it to better understand the quantitative data you have.

It’s fair comment, though, that undifferentiated ‘engagement’ is rarely a good metric.

Also, hopefully the people in charge are thinking closely about what time engaged means, for that product. I imagine breaking user actions down to intent, if possible, would help quite a bit with that.

For a product that is supposed to be simpler and save people time compared to the alternative, increasing time using the service might mean people like it and are getting more done so using it more... or it might just mean that it's getting harder to use. Then again, maybe they've graduated from doing simple things with it to complex things, and even though interactions take longer, they're still saving more time and effort overall compared to the alternative and are happy.

I guess the bottom line is I think you have to slice and dice the data a lot of ways and think about what it actually means for your product to be using that data effectively.

Absolutely. I think that’s why it’s so vital to combine both qualitative and quantitative data.

I doubt there are too many people in tech making this mistake, but without numbers there’s a good chance you fall prey to people’s inaccurate perceptions or explanations for their own behavior.

On the other hand, without those first-hand accounts, it’s all too easy to tell a mistaken story about your numbers. In particular, your users’ sentiment towards their time spent in your app is a guess, unless they tell you.

I think timescale is another crucial dimension to evaluating time-in-app, and whether it’s a positive indicator for your business.

For instance, driving up engagement has doubtless been good for Facebook’s financials in the short- to medium-term. Arguably though, that relentless focus has led to the present political climate where they’re fighting off regulation. Whether that’s an existential threat is yet to be seen (one can only hope), but it certainly casts the metric in a different light.

As you say, it comes down to the fact that there is no silver bullet metric; there’s no substitute for thinking about and dissecting the data you collect.

Author here. Great feedback and completely agree with your comments.

Taking a structure/organized approach to collecting, indexing, and sharing the qualitative data isn't something I covered here but absolutely essential to make sure that a research project scales. My experience has been that the first mile is where founders get the most lost.

Reply on news.ycombinator.com