Reducing churn with Econometrics | Hacker News Reader

Reducing churn with Econometrics

ryanleecarson.tumblr.com

38 pointsryancarson13 comments

From the title I was hoping this would talk a little more about the model used. At 45 minutes runtime, I'm guessing it's something more interesting than a basic linear model.

After taking the Stanford online machine learning course last Fall, I was very interested to hear the details about the model they used. Magic statistics apparently.

While it is good to identify users that might be leaving, it does not do any good if the intervention strategy doesn't impact overall churn.

There is also the other side of the coin. Let's say your model predicts a user as "extremely unlikely" to quit the site. If that user attempts to leave the site you may be more inclined to offer them incentives to stay, knowing they will accept a one-month 50% discount or something like that. Or perhaps their usage indicates the level of value they derive from the site, and so you can offer targeted pricing based on what you believe that class of user is willing to accept. If they user doesn't use the site as much, then maybe you could offer them an unlisted pricing tier that fits their usage profile.

And this is why I am not a business executive, it just feels like throwing spaghetti at a wall.

They probably just used a logit model, as Stata doesn't support fancy ML algorithms.

What puzzles me though, is that I don't see ANY reason for using alal of these tools together (Stata, R, SAS, etc.) and the code taking one hour to run in a big computer.

This is similar to the Adobe article on HN yesterday. Link bait for real improvement through statistical analysis, then the article undelivers.

Once we figure out if this model actually reduces churn, then we'll be happy to share.

However, you should be aware that there's no silver bullet here. The algorithm we choose might work brilliantly for us and terribly for you.

I wrote the post to get people thinking about how to use data and behavior to drive down churn.

What people seem to be commenting on is that there's a difference between reducing churn through econometrics, and an attempt to do so. The title asserts the former, that the post contains an answer.

I'm well aware. It comes down to the right combination of an algorithm and heuristics. And it's the heuristics part that is rarely transferable from problem to problem, the part you laid out in bullet points for your hired gun.

I'm just interested in understanding the methodologies, and "cherry-picking" what's applicable to my problems.

The model described in the post isn't really an econometric model, for a couple of reasons. First, the underlying theoretical model simply throws the kitchen sink on the RHS of the equation, with little to no economic motivation given for why the chosen independent variables were hypothesized to explain the variation in the dependent variable.

Secondly, only a tiny fraction of economists today actually use any statistical learning methods. Economists still don't do much prediction -- as in "predicting tomorrow, today", or even "predicting today, today" -- both of which differ from forecasting. On the other hand, microeconomic fields like game theory, labor and industrial organization often use models that are extremely rigorous in terms of the underlying economic theory, as well as with the econometric analysis, and interpretation of the results. (see: Microeconometrics)

So to the blog poster that just hired an econometrician: you didn't, you hired a statistician or computational learning fella, which from the sounds of it is working out great. So my apologies for being a stickler here, but the post didn't have any real economic theory behind it. Just some data and the results of computations on that data.

Nice, I presume the dependent variable is binary, implying a logit/probit model? I don't really get why 3 different stat tools were used, ie Stata, R and SAS... R seems indeed plenty sufficient, or even Stata.

A side comment: My current company sells to enterprises. Out of curiosity, last year I tried to look at the usage patterns of customers who left (and stayed). I had a simple hypothesis: those who didn't use the software regularly were the most likely to leave. Surprisingly, this had no correlation. It was another reminder, that in enterprise sales, low or high usage patterns are sometimes not a predictor of customer retention. It obviously depends on the industry, but sometimes it has as much to do with internal politics, salesmanship, and other things that are simply outside the scope of your code.

Ryan, Just a note but I'm assuming you're going to take action on the emails you send to users that are likely to cancel rather than just having the act of the email be the deterrent to cancel.

What's the parolhar verb?

Pardon?

Reply on news.ycombinator.com