Hacker News Reader

Comment by sgt101 | Hacker News Reader

This is a lovely bit of programming, and showcases how amazing Julia is. BUT : standard warning, pouring data into a machine and looking for the best results according to your test of choice is highly unlikely to yield a good real world outcome. You might get something that appears to be useful for a while, but there is every reason to believe that it will blow up in your face further down the line.

You are literally playing with things that you don't understand! Don't do this kids!

View on HN

Can you expand on this? If you’re monitoring for data drift and retraining every so often after deployment (not just “set it and forget it”), what are the problems that can happen?

WARNING: NOT AN ML EXPERT.

I believe this falls under the category of "data snooping" wherein you are effectively creating a model-of-models and increasing your degrees of freedom. That increased level of complexity/number of DoF means you are far more likely to over-train.

You are more likely to pick a model with no predictive power that "happened to be right" about past data.

If you're not careful, you're training your algorithm to continue promoting a bias... Perhaps one that you're not even aware of... Or perhaps someone else's bias imparts a bias on your algorithm (like turning your chat bot into a Nazi sympathizer -- an extreme example, but it could also be someone else's refusal of loans to minority groups, and your algorithm might look at debt ratio as a factor)

You can't keep assuming your inputs are always good... and while it may be okay to do some automated retraining, you really need a skilled person to revisit the inputs/outputs...

Agreed. AutoML is useful, but modeling and ensuring correct hypotheses are taken into account comes first.

totally - asking the right question is the hardest step of all, and in my experience normally the second to last one (the last one being answering it, which often takes about 20 minutes).

I agree. My experience has been that most machine learning tools help answer a question, but don't provide APIs and method to help you define the specific question you want to solve.

This is actually why I work on an open source library called Compose[0]. The goal of compose is to help people define prediction problems and extract training examples. By using a standardized API the tool can then automatically search the data for the training examples you can use for machine learning.

We're still in the early days of developing this tool, but we hope to structure and standardize this earlier step of the machine learning process that is normally ignored.

[0] https://github.com/FeatureLabs/compose

Reply on news.ycombinator.com