Comment by Dave_TRS | Hacker News Reader

Comment by Dave_TRS | Hacker News Reader

>We Can’t Reduce User Experience To A Single Number

This is the crux of the issue - CEOs do need a simple metric to summarize performance in each of their business units in order to evaluate performance and prioritize company resources. Metrics are only useful for decisions if they are easy to understand, and consistently measured. Just as Earnings is used to measure profitability, and Revenue is used to measure growth, NPS is the best they've come up with so far for customer experience. Of course none of these 3 metrics is perfect, but they still need to be measured and complimented by more detailed reporting. To better understand Earnings you look at changes in the component parts, revenue and cost. To better understand NPS you look at the mix of scores (1s, 2s, 3s, etc) and follow up questions like actual referrals, which differ by business (unlike NPS which can be applied more broadly).

The author devotes most of the article pointing out that there are areas where follow up questions would give a richer understanding of customer experience, which of course all NPS proponents would agree with.

What he could do a better job of is trying to convince us why executives shouldn't even attempt to summarize their company's performance in customer experience, just like they summarize lots of other complex activities to report to shareholders. Too many companies focus only on hard metrics like revenue and profits, which is why they find NPS a helpful way to steer the company's focus back to the customer.

Well, let's let the author try to convince you that NPS is a harmful, horrible number to summarize a company's performance on.

He would tell you that NPS is only like earnings or revenues if we allowed either to have 50% or more of their data filled with arbitrary numbers, not audited data collected from state-licensed specialists who would lose their job if it was discovered the data was manufactured whole cloth.

The author would also tell you that NPS is easily gamed and there's no checking on whether that is or not. He wrote extensively in the article the various techniques that folks can game the numbers. If this is a number reported to shareholders, shareholders should insist (No, Demand!) that the numbers be corroborated by a neutral third party that will accept liability for any errors. (No surety insurer will guarantee such a liability, for the risk of error or misrepresentation is way too high.)

As you stated, most use follow-up questions to get a richer understanding of the customer. What the author would tell you is that it's clear the NPS recommendation question taints those followup questions and diminishes their validity and inherent value. If the true goal is to learn a richer understanding of customer experience, there are many better ways to achieve it.

In other words, the author believes if executives want a simple metric that is better than NPS, a random number generator is the fastest and cheapest way to achieve it. Why bother with customers at all, if all you're going to do is squander your interaction with them on such a foolish metric.

— The author.

I think it depends on how the organisation is using NPS. If the company itself is cheating on NPS, then it’s cheating itself.

It’s like “Capital A” Agile. Sure, it’s easily gamed, but only only screwing over yourself when you do that.

I'm sure all the executives who are getting bonuses based on NPS improvements feel that gaming the system is cheating themselves.

How many executives get bonuses for a gameable version of Agile?

Again, comes down to the company I guess. I’ve never worked at a place where execs get bonuses based on NPS

Not convincing, considering executives often set their own goals.

>if executives want a simple metric that is better than NPS, a random number generator is the fastest and cheapest way to achieve it >In fact, NPS measures nothing in particular.

These types of sweeping statements aren't a helpful way for the author to advocate his broader point.

The strengths of the article lie in the more detailed points which bring to light some great examples of how NPS, and surveys in general are mis-used (and some examples that don't actually manifest themselves as real problems in daily use very often).

My thoughts on the examples chosen above:

>50% or more of their data filled with arbitrary numbers

I'm not sure what this refers to, but the way the NPS equation works, every respondent's score matters and mathematically impacts the overall NPS (each one feeds in as either a promoter, neutral, or detractor). Some of the author's own recommended questions include only 3 answer inputs also.

>Not collected from state-licensed specialists

Almost all operational data collected by companies for management reporting is not collected by state-licensed specialists, but is still useful.

>Easily gamed

All survey questions could be gamed in the ways similar to the article examples. A good executive will make sure the survey is asked in the same way of his own organization as of those of peers, and in the same manner over time. He or she won't let agents do things like cherry pick which customers to survey, otherwise the money he invests in the survey wont' actually help him run his company.

>Shareholders should insist the numbers be corroborated by a neutral third party

Correct, companies who are serious about measuring NPS accurately do often hire third parties to run surveys (see example of JD Power running NPS: http://www.jdpower.com/press-releases/bain-certified-net-pro...)

>NPS recommendation question taints followup questions

All survey questions can be tainted by preceding questions. When writing a survey it is fairly straightforward to A/B test the order to make sure this isn't a major factor.

> For some reason, NPS thinks that a 6 should be equal to a 0. Nobody else thinks this. Remember, if you worked at a company like Intuit, all that hard work to get everyone to move from a 0 to a 6 would not be rewarded. Your executive would not get their bonus. It’s as if you didn’t do anything.

This seems perfectly reasonable to me. Outcomes matter -- not effort -- and reaching 6 is not the outcome NPS wants.

Separately, the distribution will never be that narrow in practice. Once the highest rater reaches 7, NPS will start improving. The author even states herself that the input has noise, so the "everyone's a 6" argument is a straw man.

> Well, let's let the author try to convince you that NPS is a harmful, horrible number to summarize a company's performance on.

This is not how it's used in practice. Meaning: No company measures performance based solely on an NPS metric. NPS is one data point among many used to measure company performance.

There's no more sense in demonizing NPS as there is in worshipping it.

In practice, I have worked at several companies that have only used NPS as a metric.

> let's let the author try to convince you that NPS is a harmful, horrible number to summarize a company's performance on.

None of your arguments here are based on data. Do you have some evidence that a measured NPS score proved that the metric is bad? The WP link you posted to criticisms is all arguing relative merits. None of them are particularly strongly opposed, and none claimed that NPS doesn't work.

> The author would also tell you that NPS is easily gamed

Do you have data showing NPS scores being gamed?

Easily gamed and actually gamed are two completely different things. Having tried to measure NPS before, I found that 0 people appeared to be gaming the system, my customers told me honestly that my product was mediocre.

To suspect that the polls are being gamed, you assume there's something in it for the respondent, right? What benefit do you think there is for respondents to answer dishonestly?

> In other words, the author believes if executives want a simple metric that is better than NPS, a random number generator is the fastest and cheapest way to achieve it.

I hate to say it, but this kind of hyperbolic statement is having the opposite of the intended affect, it's reflecting on the author.

There seems to be a silly meme, often directed at "reductionist" engineer types that taking a measurement makes you understand less.

Nope. Feynman phrased it well:

"Poets say science takes away from the beauty of the stars - mere globs of gas atoms. I too can see the stars on a desert night, and feel them. But do I see less or more? The vastness of the heavens stretches my imagination - stuck on this carousel my little eye can catch one - million - year - old light. A vast pattern - of which I am a part... What is the pattern, or the meaning, or the why? It does not do harm to the mystery to know a little about it. For far more marvelous is the truth than any artists of the past imagined it. Why do the poets of the present not speak of it? What men are poets who can speak of Jupiter if he were a man, but if he is an immense spinning sphere of methane and ammonia must be silent?"

Reply on news.ycombinator.com