A colleague and myself experimented with some alternatives and passed notes once in a while... For certain modelling problems you can save literal gpu days by going against the grain and get better results.
Oh well...
Sometimes I wonder if people in machine learning ever look at literature.
Basic iteration schemes like the secant method (ok, 1-dimensional) have been known well over 3000 years.
Newton's method is over 300 years old.
Quasi-Newton methods (the secant method being an example) became popular in the early 1960s.
I'm somewhat seasoned on optimization methods personally, but yea it seems once people go ML they tend to um stop studying the fundamental literature that ML came from. "Online masters program learn AI in 12 weeks from nothing!". Oh okay so calculus won't be included in that... Or statistics... Or... Yep it's going to be scikit learn notebooks ...
> Baseless empirical result that probably was p hacked
This to me seems like the biggest regression in science. It's all heresy which is very hard to re-produce or learn general lessons from. It feels like disparate social science methodologies are being used to study math.
Nobody is going to look back and benefit from these papers. I often bring up to ML folks limitations proven in the book Perceptrons and wonder how their models differ. I have never gotten a response.
What field did you move to?
I float between a few technical fields. Some in natural science, computer science, data science hybrid roles, data bases/engineering, etc. Not a jack of all trades, nor a master of none. What I do have mastered isn't something people hire for, so basically I am an averagely smart person who will take any job and figure it out to pay the bills.
At home though I play with all of the areas of creation I can get my hands on. I guess I am just in the field of discovering new things and making things.
They don't.
A few other things they seem completely unaware of:
- other ways to represent functions besides neural networks (harmonic analysis, polynomials, etc)
- other models exist besides neural networks. ie. if you can model the problem with a simple equation you can just optimize that.
- polynomial regression.
Someone who has read "numerical recipies" is probably more capable in solving ML problems than an "ML software engineer".