One of the ML mistakes that will hurt most: not collecting the right features | Hacker News Reader

One of the ML mistakes that will hurt most: not collecting the right features

twitter.com

56 pointstosh22 comments

One of the lessons I'm learning in my work right now is:

Anyone can throw a mess of features into a huge hog of a model and get some sufficiently accurate input. But with feature engineering, you essentially absolve the computational model from needing the vast majority of the capacity in the naive case, instead "pre-computing" those transforms which would otherwise be learned by the model. With the reduced need for capacity, the ability to refine the model for interpretability's sake (and optimization stability) also improves.

Of course part of success is also collecting the right data, but focusing on the right features is a slightly more nuanced and focused way to look at it IMO.

This doesn’t seem like good advice. As a human, the data might not be engineerable by you. The data might seem like it is, but you may end up losing information through your feature engineering. The combinatorial power of NNs is so great that it’s almost always better to give the model as much data as possible and let it figure out which parts of the data are important.

You're right, my advice may be biased. I work on physical systems operating under tight control and near equilibrium most of the time.

I've spent some of my career doing Data Science consulting, and easily 80% of my time is spent on finding, cleaning, formatting, and studying data. Building the model is almost trivial if you take the time to get your data in the right shape (i.e. removing noise and unsalient features, and making sure you have every usable bit you can get)

The best datascience projects are ones that started collecting good data 3 months ago. The hardest thing I've had to do is convince engineers to involve DS people in the design phase of their work. Initially they just don't see us, or see us as an annoying extra stakeholder. But it's more annoying to go back to already deployed code and add data collection to it and then waiting, so they learn fast. Anyways, like the OP's link says, there's no magic modelling work that can bring back those missing features and data.

I’m a product manager - The company (medium sized public company) I was at previously asked me how we could better utilize machine learning in our products.

I spent a month writing and implementing a new company playbook on a building products with data science.

It boils down to including data science early during discovery (In part to help identify what data we have available up front), and viewing them as a squad stakeholder rather than a team for one-off work. It also pushed everyone to start thinking about the reliability of data for every product we release.

Some of the databases I’ve seen at companies is just an absolute abomination.

With the explosion of deep learning, I don't want to say that feature engineering is a lost art - no at all - but many of the benefits of DL, such as automatic feature extraction, is certainly something to be considered before just jumping into a project.

I've seen examples of where good and careful data collection in advance would have solved the problem, but instead, someone decided to transform the (easiest) collected data into a format suitable for DL - and spent months trying to optimize and tweak the DL model. Because their rationale was that it's much easier to just fine tune an out-of-box DL / transfer learning model, than spending ages on extra planning, data collection, feature engineering, and what not.

"One of the ML mistakes that you will hurt you the most: not collecting the right features. If the information you need isn't in your data, you won't recover it with a better model."

The word 'information' is doing some heavy lifting here. There is nuance - what information could be learned or computed from the data? We can only really guess. When we're wrong about this, then we will design models imperfectly.

Is there enough information in internet text scrapes to solve unseen high school mathematics word problems? The feature engineers of olde would have said no. But there is, to a surprising degree.

The way of speaking about "data" being an undifferentiated mass of bits, from which you get "useful information" is pretty well-worn, widely observed piece of rhetoric.

(If it's wrong, the scope of the mistake reaches far beyond this article.)

This approach of data maximalism seems to work well primarily for NLP and CV. However, there are lot of domains where throwing every feature without some consideration of information will not get you good results. To take a simple case, you are asked to model (forecast) the sales of an item at a retail store. Jamming in features like - color of people shirts, average length of their nails, the duration of traffic signals in front of the mall, the length of the store name etc.. will not lead to good results. In contrast, seasonality, prices, promotions, weather etc. will most likely lead to good results.

Why does the second set of features work while the first doesn't? I like to think of the second set having a model of causality, to figure that out you need some domain expertise and experience. Without that expertise, the feature set of non-causal factors is almost infinite.

I think the real biggest "gotcha" is not understanding where the control loops are in your data.

The classic example is the "air conditioned room" thought experiment. Imagine feeding an AI model a dataset with a bunch of datapoints like this

Room hot | AC on full blast

Room cold | AC off

Room warm | AC on 50%

Then asking the AI how to make a hot room cold. The AI is very likely to say "turn off the AC" because "AC off" is a feature of cold rooms.

Maybe we think that if we give the AI more information, it could do better. Lets imagine that our room has a window and an oversized AC. We'll now be feeding it info about the source of heat as well as the source of cold, so surely it will get the right answer this time:

Window open | AC on full blast | Room cold

Window closed | AC off | Room cold

Window 50% | AC on 50% | Room cold

This dataset is going to have the AI thinking that neither the window nor the AC impact room temperature, and that you might be able to close a window by turning off the AC.

The key thing to understand is that control loops change the "physics" of your system and that knowledge about the control-physics does not transfer to the uncontrolled-physics.

Machine learning can deal with this very easily by incorporating time as a feature.

That’s the point: without knowing that time is an important feature your ML model will come to the wrong conclusion.

I don't think that was the point the OP was making. I think they were trying to argue that ML cannot learn control loops or maybe that the system needs to "understand" physics (Not sure - it wasn't really clear to me).

The fact there is a control loop here doesn't actually seem important if you represent the features correctly.

I'd argue that data representation is actually what you need to get right (ie, in this case the behavior in the time domain).

The correlation exists. If you see that the AC is running a lot then you can predict that the outside temperature is warmer.

What machine learning algorithms even try to determine causation?

Strongly agree with the author. A logistic regression built on top of perfectly crafted features will out perform more advanced approaches with crude data.

I think it’s too much simplifying ML. If features are so easy to find yes you can write a rule based system and it works. But most of the real life problem is not easy.

The scary thing about ML in practice is that it seems that it works until it does not work at all. Then people freak out and dismantle the entire ML team.

Isn't this just "garbage in, garbage out"?

Yes, but the statement provides more precision on what garbage is.

The weakest link is always the human factor:

1. data

2. algo

Funny he would say that because a few days ago in one of his tweets he was calling the stock market bottom for this year. I replied that he was using the wrong inputs:

https://twitter.com/melling/status/1537562264387166209?s=21&...

People make the same mistake in their decision process.

I was going to add that his model wasn’t explainable either but only so much geek humor…

Reply on news.ycombinator.com