Predicting Churn: When Do Veterans Quit? | Hacker News Reader

Predicting Churn: When Do Veterans Quit?

gamasutra.com

67 pointsukdm32 comments

You can do a very, very similar thing for non-game SaaS services, too. (Hubspot calls theirs CHI -- Customer Happiness Index. It attempts to predict who is going to quit based on backtested statistical models. In practice, you can get almost sickeningly good results with heuristics that take ~3 lines to program. You can also then save those accounts by something as simple as "Have the sales team email them and talk" or, in my recent experience, "Send them an automated email reminding them that the software exists.")

As a user, those emails have sometimes had a negative effect for the company emailing me. Something they send me reminds me that the program exists, that I'm still paying a monthly fee for it, and that I really don't use it very much at all.

But in all fairness, I doubt anything they could have said or offered would have changed my mind at that point. It was just a matter of when I cancelled, not if.

I think your last sentence is important - you were already gone, it was just a matter of time.

If the numbers work such that losing 1% of customers means keeping 3% i.e. the email prompts them to leave/stay respectively, then it's worth sending.

Getting rid of low-value users - in this case those not playing - can have long term benefits of focusing the business on getting more custom instead of getting comfortable on revenue that may disappear at any point.

Stochastic Solutions have several papers about this problem: http://www.stochasticsolutions.com/papers.html

In a subscription business who can be saved, who will leave regardless and then the tricky group of people who will leave if they get a sales call?

This looks really interesting - thanks for sharing!

The reason for emailing you could also be to find out why you aren't using the app. Even if one ends up closing the account as a result, the information gained is valuable for improving the product.

Alternately, noting them but not sending them an email at all.

Much like a gym membership, the people who don't use it at all but feel like they should someday (as long as they're not nagged about not using it) are one of the most profitable subscriber segments. Sure, they'll leave eventually, but they're not using your service anyway, and until then it's free money.

Patrick, just to be clear: Are the heuristics you're advocating here just something along the lines of "E-mail this user if they haven't touched their account in X days" or is a more in-depth approach required?

My previous employer knew I was going to quit long before I told them.

But then, the fact that I was sending out resumes was probably a huge hint. I started sending them when I was a little disgruntled. Eventually, they heaped crap on me until I got serious about it and found a new job.

Not once did they attempt to do anything to stop me, and asking them to pay me what I'm worth was met with, "Can you wait a year?" This, despite the fact that every review I ever got was great. Not just good, great. The only complaints I got was that I was too quick to answer questions literally. If someone asked if something was possible, I told them. They wanted me to read their minds and ask the question they should have asked, instead of the question they did.

I eventually learned to do that, even, though. I have to admit it made things smoother. But jeez was it a pain.

The best response would be to offer a reason for people to come back after they leave for a week or so.

The worst response would to offer a reason for people to leave for a week or so if they weren't going to already.

If the company basically offers an incentive for not logging in, word will get around very fast. Then players have an excuse for taking a break, that they are gaming the system to earn an incentive.

Penalizing players is ineffective too because you want to welcome back your wayward customers instead of starting to burn bridges ("I'm going to throw out your stuff! Okay, I'm putting it in the trash right now!")

So unfortunately, I think all a company can do is have emails and community managers get in touch. Of course, that encourages players to stop playing for a week whenever they want to escalate a customer service issue. It's like you can't win.

I discovered at Blizzard that guild features were enormously strategic. When players get into an active guild, especially with people they already know, it becomes hard to leave. The glue that keeps players put is social expectations.

That's a really interesting approach. Basically mining a couple of key features from the data set, and plugging it into a supervised learning algorithm. The really cool thing is the simplicity of the features used, just daily activity and daily playtime.

The write up of the methodology is very clear, but I'd love to see some more description of the results. Ninety-five percent accuracy is a pretty bold claim, and I'd love to see some ROC curves to back it up!

The high accuracy surprised me too.

To compute their accuracy, their methodology seems to require determining whether someone is a veteran user, and having a clear quit time for them (otherwise the user can't be used for training or testing). Maybe after you make these determinations, the resulting population is easier to deal with.

This is a bit scary. Imagine your employer using machine learning on your git commit patterns (or swearwords per comment) to check when you might quit.

I can imagine it. Why is it scary?

Would it also be scary if your manager observed your attitude/emotions through personal interaction and deduced you might want to quit?

I'd be all for a tool like that, assuming it actually worked and it was used in a productive way. Imagine if you're getting tired of writing RoR code all day and pondering quitting and all of a sudden your boss comes up to you and says "hey how would you like stop coding Rails for a while and join a team that's about to evaluate Clojure for an upcoming project". That's just the sort of thing that could make everybody happy.

The problem is, at least in my experience, the ratio of people who would use it for good vs those who would misuse it would be very bad.

Of course, it's also the employee's fault for getting into such an environment in the first place, but sometimes you just need the job.

I use ManicTime Tracker, if my boss got hold of all the data contained in there, he would probably consider firing me - I average more than 2 solid hours of surfing the web on any given workday... wait, I have the exact data :) , that's the point!

I was at my work PC for 1059 hours so far this year.

Of those, I spent 325 hours on Firefox and 95 on Chrome, that averages about 13 hours a week of web surfing (probably 10 hours procrastinating or reading and 3 actually researching problems).

I also spend a shocking 8 hours a week reading and replying to mail (to be fair, we don't have any bug tracking or project management, so mail becomes both), and 5 hours a week on SQL (I do a lot of querying and reporting).

I'm less than 4 hours a week actually on a development environment, split between VB6, .NET and Forte4GL (our ugly legacy system).

And I was hired to be a "systems analyst"... And that's only time spent at the PC, it doesn't count time wasted on meetings and stuff.

This is a real-life example of why they say that on a large corporation, you only do actual work 1 or 2 hours a day (as opposed to a startup where you might do code or programming-related stuff 6 or 7 hours, I hope :) ).

It's really depressing to put it in numbers. Fortunately, I'm going to quit next year and dedicate full-time to my startup (which is just getting started right now :) ).

There is no doubt extensive tracking data would be just another item in the micromanagement toolkit for many companies. What is interesting is that companies are succeeding without that kind of analytical data on programmers and IT staff, so you can imagine that having that data would only increase incidents that make people want to quit and do nothing to help with management or productivity.

> Of course, it's also the employee's fault for getting into such an environment in the first place, but sometimes you just need the job.

That someone ends up in a shitty workplace isn't their fault. You can fault someone for actively making their workplace worse, but sometimes you end up in a bad place, either out of necessity or just plain bad luck. This kind of attitude towards workers in the workplace only hurts people's ability to find meaningful work and places to work.

It's called Short Timer's Syndrome. When you know you're leaving, your dedication and work ethic plummet. You don't need a lot of statistics to notice it; you just have to be looking.

Many managers are too busy looking up the management tree, sucking up to superiors, working their own career to actually manage their subordinates. If you're expecting to leave then you're in luck, they won't notice. But that's probably the reason you're leaving too!

I didn't know it had an official name :) . There are papers and everything (example: http://www.opus12.org/uploads/O12-SCI-V02-N01-P30.pdf )

I guess surfing the web is a form of short-timing - I can't actually leave early or come in late, since I'm heavily penalized for those, like any self-respecting bureaucracy, my company equals time at the desk with productivity, and heavily penalizes lateness.

You're assuming that managers may actual care about the intellectual and professional needs of the staff instead of fulfilling their bosses goals. (Or maybe I'm just jaded and need to find some greener pastures... :) )

Most bosses in the US would use these metrics to more quickly replace those who were about to quit.

Only if the boss has replacements lined up, which is rare in professional jobs.

When I took a HR course (for MBAs), I learned that Human Resources is supposed to have a replacement roster (it's part of their job), I guess that's not common in practice, but they should.

For me the difference is between expected and unexpected. I expect bosses to observe my attitude/emotions through personal interaction and even review my commits by hand (it's even mandatory where I work.) I do not expect them to compute extensive statistics on what I do in the repository and draw deep conclusions from that, it's somehow too close to what I consider private. It's obviously subjective.

Well, now it is expected. Welcome to the post-privacy era.

A manager who has time to log all my actions throughout the day then study them to try and guess whether he's doing such a bad job that I'm thinking of quitting has probably answered the question before they even start.

If your boss needs machine learning to see that you aren't happy, there are much bigger issues.

Google does something like this, apparently:

http://online.wsj.com/article/SB124269038041932531.html#mod=...

The article cites customer service interactions as the obvious (and obviously effective) remediation step once a player is identified as likely to quit in a few weeks. I'd like to know if this is really true. What if the game's fundamental problem is that players become bored over time? What can a CSR do to fix this? Probably not much. The analysis shows that changes in gameplay frequency are far from a random walk and do indeed predict future declines, but it doesn't offer much in the way of clues for how to keep customers engaged. What if it could be shown that a breakdown of a player's in-game social structure (I'm not a gamer, so I'm being vague) also predicts churn? Perhaps helping the player to find some new relationships would be effective. Is game play repetitive? Maybe decreasing frequency of novel experiences predicts churn? This would be good feedback for game designers. Predicting the future isn't of much valuable if it isn't actionable.

This is really interesting.

I can think of great possibilities for using these methods to analyse the behaviour of users in any number of online services to identify the ones that might need some form of out reach to help them out and keep them as users rather than loosing them due to the problems they are having.

There was an article with a similar goal previously posted to Gamasutra from altdevblogaday, but it's got data and software for you to play along: http://www.altdevblogaday.com/2012/08/10/business-analytics-...

I bet you could apply this method to other participation-based business. One that springs immediately to mind is gym membership.

Reply on news.ycombinator.com