Hacker News Reader

Comment by minimaxir | Hacker News Reader

With the current low cost of cloud computing, there's no reason not to just try everything and see what happens (which is why AutoML has become more popular).

It's more pragmatic than trying to rationalize which framework is "best" for a given dataset, as the results are often counterintuitive.

View on HN

On the contrary, I think this is one of the biggest emerging blockades to progress in ML/AI research, especially in academia. It has always been more cost-effective to run ML algorithms on consumer HW such as GeForce GPUs and gaming CPUs. It's frequently even faster than contemporary cloud offerings when the consumer HW gets ahead of existing enterprise HW. And it's so effective that HW companies starting changing their EULAs and crippling previously available aspects of APIs to herd AI back into the datacenter where they seem to think it belongs.

And that IMO is a reinvention of the "Walled Garden" of academic HPC (ask any grad student begging and pleading for supercomputer time) which has always sucked and its new commercial incarnation is even worse because it's unclear how to get commercial cloud time on government grants.

OTOH it's fine for large shops like OpenAI, DeepMind, AWS AI, FAIR, MS Research etc because they have deep deep pockets. So if you're content with most future groundbreaking research coming from a small tribe of market leaders, well great, but I suspect innovation is already slowing down because of this.

this approach is also very inefficient- autoML takes hours searching for parameters when you could build the same model manually in a fraction of the time.

It depends on what algorithms you use in AutoML. If you decide to use simple algorithms: logistic regression, decision tree, random forest then you will have a simple model very quickly. Using Neural Networks in AutoML requires much more computational resources.

There's no reason you can't do both: prototype a simple model to get a baseline performance, then use AutoML to fine-tune it, plus you now have a value to sanity check against.

Those hours of hyperparameter search aren't blocking. You can do other things while it's searching, or do the search when not actively using the resources (e.g. overnight).

if you know optimal hyperparameters from the beginning, then yes, you can build such model manually. But in most cases this is not the case.

I don't like such brute-force approach. Even if you have low cost of computing the number of possible combinations of hyperparameters is huge! Google Cloud AutoML Tables solution cost is 20 USD per 1 hour of computing (I guess that's because of inefficient Neural Architecture Search algorithm). Running few ML experiment can easily end with huge bill.

For independent ML hobbyists, sure. But if you're a business which can get a positive ROI from it, it's well worth the time.

It depends. Complexity creeps up easily in ML systems. A kitchen-sink of gazillions of algorithms (and code bases) in an ensemble creates a very brittle system I wouldn't want to deal with.

Reply on news.ycombinator.com