Finding your One Metric That Matters using SQL

Useful post. NTILE operations are also useful for evaluating the results of 50/50 tests against revenue or other metrics where your data fits an exponential curve, or where outliers tend to skew the performance of one group incorrectly. After normalizing your datasets so that each group has identical numbers of users, a graph comparing the distribution of your metrics on a percentile basis can make it trivially easy to visually evaluate the results of a test.

As far as using NTILE to find the one metric that matters is concerned, it's a useful tool, but the challenge isn't in structuring and executing SQL queries so much as it is deciding which metrics to focus on and attempt to correlate to user retention.

In my experience there can be hundreds of metrics to search through, and correlations between your metrics and user retention are not always obvious at face value, or worse, they might represent a tautological consequence of retention without being the cause of retention (ie, the user has lots of badges because the user is active on the site, but they are not active because of the badges). In these cases, you risk investing valuable development resources optimizing for something that will have no impact on the bottom line, when that time might have been better spent trying to identify novel metrics and getting them into the database for analysis.