NPS doesn’t say anything particularly useful
cranberryblog.substack.com
cranberryblog.substack.com
Also, I find that many of these articles have somewhat of a caricaturish view of how NPS is used in the real world. "It's not 'oh no, our NPS score is down, let's hire more people to focus on customer 'happiness'", it's instead that NPS can be like a canary in a coal mine - an unexpected drop in NPS is something to investigate and get more information on what went wrong, usually from more detailed/free-form questions in the NPS questionnaire.
As a software engineer I love reading our NPS reports, because they always have great tidbits of info that are often difficult to deduce just by looking at standard analytics.
There is avsolutely no science linking performance and NPS. Just pseudo science.
There is no underlying science behind NPS. It was an arbitrary measure with no research or model behind it. Also, it doesn't work very well, in large because because there is no strong correlation between NPS and anything else, because - as this article makes clear - NPS is an extremely noisy number. Look at the graphs - you can have enormous variations in NPS, with zero change in underlying customer sentiment. Obviously that can't correlate with anything useful.
I think what you're doing is conflating the idea of "asking how likely people are to recommend your product" with NPS. The former is a decent idea, and it does have some (limited) science behind it! There really was a study done that really did find that question was an excellent predictor of many key results. (...and some others that found it was not, but hey, it's something.)
So yes, asking people the question and reading the results is probably a good idea. ...but that's not what NPS is.
> an unexpected drop in NPS is something to investigate
It really, really isn't. And I would challenge you to point to any evidence to the contrary.
After a quick search it seems that NPS originated in a Harvard Business Review article, which I don't consider a credible source of scientific results. The scientific papers I'm seeing mostly seem pretty skeptical, judging from the abstracts.
> a caricaturish view of how NPS is used in the real world. "It's not 'oh no, our NPS score is down, let's hire more people to focus on customer 'happiness'"
That sounds exactly like what responses at many companies would be. There's often much nuanced discussion about how/when/why to adopt certain metrics, but once they are they become sacrosanct and any deviation from "good" results is a crisis needing immediate action.
(And I have plenty of opinions on how useless KPIs have been in practice for me and the places/teams I've worked. They're hard to get right unless everyone in management is on the same page about what a clearly defined business goal is, and what an actionable step is, and what a strong metric for determining whether the action met the goal is.)
What you love is the customer feedback--this can be accomplished in multiple ways and you don't need NPS for it.
A proper NPS asks only one question and requires only a single click to answer -- eg "Scale of 1-10 how likely are you to recommend <product> to a friend?" > click [8] > done.
There's a free-text "Tell us more" field that is totally optional.
This means users actually answer it. NPS feedback is the closest I've ever seen to being a clean sample.
By contrast:
- Everyone hates an n-question "would you like to fill out our survey" feedback form. Any data from those is going to be skewed, because only a certain kind of person fills those, and usually only if they're angry enough that they want to vent.
- Feedback from a passive "Submit feedback" button in-app is also skewed. Maybe useful as a bug report mechanism, but it won't tell you what's making your happy users happy.
- Proactively reaching out and talking to/observing users is obviously good, and nothing ever substitutes for that. But you want bulk data, not just anecdotal evidence.
- Most importantly, NPS really shines at telling you what's making your almost-happy users almost-happy. When someone clicks [10] and writes "I love it", that feels great. But when someone clicks [7] and tells you what's annoying them, that's often extremely useful.
- NPS does a great job priming people to give useful feedback. By forcing you to pick a single out-of-10, you just did a quick mental accounting. "9? Nahh, <reason>." Then you click 8 and write the reason.
--
NPS was popularized by that famous HBS study where they found it correlates well with product growth.
I wonder how much of that result is simply because it's one of the least obnoxious ways of sampling user feedback, and therefore produces clean data.
On the other hand, like many, I probably tend to follow something like the XKCD star rating levels (https://xkcd.com/1098/) for products. And for employee surveys, I generally answer somewhere in the mildly positive range to most questions.
I'm not sure I've ever answered an NPS survey but, for most companies, I'd probably be somewhere in a similar range, even for companies I'm perfectly fine with most of the time.
The more complex the transactions/products the more reliably you'll have areas of gripe-age. A book or movie is rarely 5 stars for me. A USB cable pretty much works or it doesn't.
The only way this would be a meaningful effect is for a dev tools or similar product where your audience is the HN audience.
The reason why NPS exists is because it's an easy number to calculate that sounds convincing, and manager-types love numbers without having to think too hard about how they got them.
1: https://web.archive.org/web/20200716065914/https://pdfs.sema...
But management-types love little numbers like NPS, so it usually gets done anyways, especially in big orgs. And then it goes downhill, because PMs and leads are incentivized to optimize for the particular number their management chain tells them matters, and they game it because that's how people work. Later on, frustrated engineers and PMs who aren't a part of that game wonder if they're the crazy ones because they see very real customer frustrations brushed aside by an org structure that doesn't seem to care much about what users actually tell them anymore. Or they say they do, but never incentivize the rest of the org to address issues coming in through verbatim feedback.
Maybe, someone with the title of VP eventually wonders why a competitor is doing really well, looks at verbatim feedback themselves (it's usually been given to them already but they forgot about it), and then realizes they've been steering a big ship in the wrong direction, and causes chaos all kinds of chaos in the process. The sycophants in the org all line up to agree with them and declare they were right all along (and get rewarded at the end of the year), line managers and their reports are left confused because of the jolting priority shift, and people who might have felt vindicated by the direction shift are wondering if their leadership are actually fit to be leaders or not.
Is that all the fault of NPS or other numbers? Of course not! But they're an easy and generally accepted way for bad leaders to hide their bad leadership.
Source: Worked in multiple startups that IPO'd using this as one of their 'growth strategies'
This is precisely what happened at my company, hah.
My only experience of NPS is when I worked retail/sales in my first job, and it absolutely sucked in that context.
Less “great tidbits” and more “I just lost 30% of my commission this month because of something 100% outside of my control”.
It's nice (but a bit too late) to know that your business is screwed because your customers left, it's another to know that your business is about to be screwed because your customers are on the verge of leaving if an opportunity appears.
The criticisms of NPS pointed out (that it's a measure of a high-variance metric) are fair, but the conclusion is not.
Ultimately if it matters, it should have to be an input to the cash flow time series -- mediated by churn -- no? (Article means to frame free cash flow as a series rather than a point-in-time value).
If you're interested in forecasting, as a company, you should know the things that mean your customer is not doing well within the specific context of your specific business. E.g. in software, you should have a customer health score that's built up from product data. Or you could ask simple questions that are easily interpreted, e.g. asking "how would you rate X's value for money," or "how satisfied are you with X," or even "have you recommended X in the last Y months." These things have a cleaner relationship to future business metrics and a tidier interpretation.
I’m skeptical as to its usefulness even for this, but most companies I’ve worked for use it as a general KPI which is even more aggravating.
I imagine you are annoyed that they optimize for it. And yes, it's not reasonable to optimize for it. Why do people optimize every KPI?
Because if it's not worth optimizing around, how key can it be?
For example, you can satisfy it, you can alert on behavior, or you can use it as a control (but ok, maybe this one makes a two number KPI instead of two KPIs).
Esp in early days of B2B products, it's hard to get that because you don't have the volumes and velocity of b2c nor a good way to detect and attribute viral activation. If you are in a startup and not riding the channel of some megacorp, even more so. Alternatives like signups or other activation checkpoints, or say qualitative interviews, are also interesting, but even more spotty. (Ex: startups raising based on GitHub stars.) We don't do NPS as we have our plate full with known funnel holes through less annoying data collection methods, but as soon as we are happy with the baseline funnel, that's the simple next step.
They're thinking in terms of "how satisfied was I", which is a different scale entirely with "would you recommend the product". But the NPS question, even when explicitly explained, reads close enough that they'll answer it with the "how satisfied am I" answer,
NPS might have worked in a context of a professionally proctored focus group where everyone understood the question and discussed it, but I'm worried you're losing a lot of information when you turn it to one digit on a keypad.
There seems to be this idea that KPIs are evil and interpreted in vacuum. It's rarely like that.
Not just KPIs, but the Hackernews crowd seems to apply the same to management and corporations as a whole.
My advice to friends looking for a job, ask them what their NPS is. The leaders in NPS are leaders in employee engagement. If they don’t know, or it’s bad and your role isn’t to make it better, look elsewhere.
Never mind that anything below X is considered neutral or a detractor. You asked a user / customer/ etc to rate you on a scale of 1-10 and then tell you why.
Want to know what you’re truly screwing up on? Take the feedback on your 1-6 scores seriously and you can find the low hanging fruit to take a product from mediocre to a great user experience.
That’s the true NPS value. It’s all about how you handle the feedback.
I noticed that people are concerned about this when it was not implemented correctly, i.e. as a complimentary question in the long questionnaire or as a follow-up question at the end of the journey.
The NPS collection mechanic has many advantages:
1. The mental entry point for the user is very low - just 1 question.
2. It is very useful to track user feedback over their lifetime (ask every 6 months) and actively resolve issues if NPS shows it.
3. Another great thing is not a numerical feedback, but a written answer that gives a lot of really useful ideas for improvement.
4. And finally, you don't need any specific context to show the question, so it can be embedded in almost any stage of the user experience.
As a single metric Net Promoter Score is OK. It's extremely easy for customers to answer. So there's volume in the number of answers. You can learn more by segmenting different ways—Users vs Admins, Enterprise vs SMB, Verticals, etc..
The company I currently work for puts a lot into NPS, not necessarily the metric, but the process. Everyone that leaves feedback gets a follow-up email and interview from someone in Product or User Research. That feedback is then organized in Productboard where it is grouped with similar requests/issues. Overall those interviews heavily weigh what features we refine and build.
For some background, I previously designed an NPS tool and have written about NPS pretty extensively.
One of the biggest problems with it is that an average person, unaware of what NPS is, doesn't understand that giving any rating less than a 9 is essentially giving a rating of zero.
If I have what I would consider to be an average interaction with a business, e.g. I just buy something and leave, no need for support, no problems to deal with, etc., that seems average to me. Based on a non-NPS understanding of a 0-10 scale, I'd say that's what, a 7? But the business now considers this as a failure on the part of the person that helped me at that store.
This is why sales people and phone reps are constantly now asking you to give them a 10/10 rating if you receive a survey, because even if they literally just took your money and handed you change, their jobs depend on you acting as if you had a heart attack and they saved your life by performing CPR or something.
It's honestly a terrible system that produces no meaningful feedback for the company and causes employees to do whatever they can to game the numbers. All you're measuring with NPS is how good your employees are at juking the stats, nothing more.
I like to look at this kind of thing as the one of the dark sides/dark patterns of using data for decisionmaking.
Qualitative measures are important, as is maintaining as much humanity as possible, to have a balanced and healthy culture. Being solely metrics/data driven can lead to cold, heartless, damaging culture (might be efficient or make profit, but very dehumanizing).
“The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide.”
https://en.wikipedia.org/wiki/Goodhart%27s_law
My favorite story related to this, because the managers thought LoC would be a good measure of productivity, is "-2000 lines of code": https://www.folklore.org/StoryView.py?story=Negative_2000_Li...
Sometimes they require writing something to justify a bad score (but not a good score!)
I would get so many 10s for the rep (which does not count at all) and 1s for the company which would obliterate my stats. And, only the last person who spoke to the customer was rated, so this encouraged meaninglessly transferring people around like a hot potato.
Number of times I would get a customer who had issues for weeks, and then I take a look at their issue, resolve it in 15 minutes, they would be ecstatic on the call and then the NPS survey comes back, 10 for me and 1 for the company. Then I would get a tap on my shoulder from the supervisor asking me to explain the detractor.
I almost got fired after about 6 months due to my NPS being too low, until I made friends with one of the vets, and one evening in the pub with couple beers too many in him he explained to me that the only way to survive is to game the system. He told me how to crash the call client to prevent it from sending surveys to angry callers, how to transfer people to infinite hold queue which does not result in survey when they hang up, and how to trick the system into thinking that I had and inbound call when I did not (which gave me time to actually do my work, like performing relocations and resolving complex provisioning issues since any time not spent on inbound call was considered to be not adhering to schedule). I went from less than +40 NPS to +90 NPS in one month, so most of the NPS feedback was fake anyway.
Well, at least its not a negative number!
Companies that are leaders in NPS are leaders in employee engagement. This post sounded pretty frustrated with an employer/contract…
NPS is a measure of past experiences, it isn’t a GPS navigation system to improvement. It is also 1 question, so it is the most simple to collect. However even this gets screwed up all the time. Interfaces such as a telephone where 10 can be measured as the first digit 1, or PIN pads where they are a nuisance to a customer when in a hurry to pay and missed when paying with NFC.
Understanding negative NPS is like understanding a traffic ticket with no other data. Why did I get a traffic ticket? Well, what did or didn’t you do and how fast were you doing it? Were you a jerk to the cop? Was the cop stressed by something else that day?
To be a leader in NPS is an organizational effort, not the result of divining an insight from a few survey answers. Ideally, customer journeys maps have been developed and problem areas are self evident before an NPS score arrives.
No, IRS, I would never recommend your service to friends or family. That's not really indicative of anything.
Assuming they gave a shit, that survey response would have been useful to the practice.
They should be asking doctors. And if they want your point of view, they should ask something minimally reasonable (like, did you have any problem using it?), not some crazy hypothetical.
> How likely are you to recommend Windows 10 to a friend or colleague? Please explain why you gave this score.
> I need you to understand that people don't have conversations where they randomly recommend operating systems to one another
> measures customer experience and predicts business growth
Ask people every so often how they're feeling about the product. Let them pick a thumbs up or a thumbs down. If they pick thumbs down, say "we're sorry you hear that. Please let us know what's bothering you", and display a little text box. Carefully read every comment you receive.
Asking unhappy users for feedback is a great idea. The 0 to 10 scale and - especially - the bizarre calculation NPS does with it - is statistical malpractice that just isn't mathematically capable of doing what NPS supports claim.
I disagree. On the contrary, it is good survey practise to ask yes/no questions with a scale, to get information on intensity of opinion. (Often this is a scale 1--7 ranging from "strongly disagree" through "neutral" to "strongly agree", but there are other alternatives of which NPS is one.)
Then you might think only the bottom half of the range would formally be considered "detractors", and that would (as far as the evidence goes) be accurate. However, we're not looking for how many people are detractors today -- we want to know how many people are at risk of becoming detractors in the near future. That's what makes NPS a somewhat leading metric. And that's also why you see the sleight of hand that counts answers up to 6 as detractors.
I don't know what "mathematically capable" means in your context, nor what "NPS supports claim" in your experience, but the number absolutely makes sense.
NPS has had a strong prediction of stock price (I learned this morning).
https://www.marketwatch.com/story/this-surprising-investing-...?
Of course, it’s just opinion. But maybe there’s some use to NPS after all.
1) it's harder to game and/or bullshit. This is a massive issue with surveying customers and then using their responses to feed back into internal performance goals & rewards.
2) because it has grown to be such a common customer experience metric, it's possible to benchmark against others, which isn't really possible with question types that are very specific to your business
I'll agree that NPS is overrated, mostly because the Net Promoter organization and survey companies have spent large amounts of time and money pushing it as The One True Metric for decades, but it still has its merits and its place.
I don't have access to A/B testing of a large business to understand how things affect responses. But I certainly intuit that there is a significance in when/how you ask someone for feedback. e.g. If you ask your question directly after the checkout screen you'll get a significantly different response than if you survey people randomly who visit your website, or email customers a set period after making a purchase/ upon transaction completion.
I've seen all these, which brings me to the second part, how do you benchmark against a competitors data, and why would you trust a competitors data?
Except the alternative isn't to stop tracking NPS the alternative is to use unstandardized customer survey metric's, this is worse because you have no true industry comparison & can be gamed more easily since they define the formula!