Hacker News Reader

5) Take a random sample of 50,000 records and get 90% of the business intelligence value in about 2 seconds.

If you're fitting a model with only 70 degrees of freedom then analyzing 150M records is a complete waste of time.

This is absolutely not true for insurance data, where the task is to predict expected losses per policy and (in any given year) perhaps only 1% of policies will have any losses at all. Even if your statement were true, this sort of analysis has nothing to do with business intelligence. The goal is to minimize adverse selection in a competitive marketplace. There is no such thing as "good enough". (If there were, I would be out of a job.)

Try stratified sampling. Removing records without claims only increases the variance of the denominator which is much less variable. You actually can eliminate the majority of the data and find results that are the same to several decimal places. Note this only works with very large datasets without extremely high cardinality variables.

That said, 50000 is too few. For a dataset of this size, 20 million records is likely more reasonable. The actual answer depends on the variance of the individual predictors and their correlation with each other.

Reply on news.ycombinator.com