Hacker News Reader

Comment by bobosha | Hacker News Reader

>"Outside of neural networks, GPUs don’t play a large role in machine learning today, and much larger gains in speed can often be achieved by a careful choice of algorithms." [1]

Scikit learn espouses a non-GPU approach. Perhaps the performance gains by using GPUs aren't that significant. Has anyone tried SVMs (or for that matter other non-DL classifiers) + GPUs?

[1] http://scikit-learn.org/stable/faq.html

View on HN

XGBoost has GPU support. It is recent: the paper was published in 2017.

http://xgboost.readthedocs.io/en/latest/gpu/

https://github.com/Microsoft/LightGBM

LightGBM also has GPU support for learning decision trees.

Note: LightGBM paper https://arxiv.org/abs/1706.08359 cites XGBoost paper (reference 16).

Anyone has more insight into why? if I try to grid search hyperparameters for a random forest takes ages in a single machine with scikit-learn. I only ever see GPUs in neural networks. Is there some acceleration for non-NN machine learning algos?

As an aside, you should generally avoid grid search in favor of random search or an evolutionary algorithm. Grid search can waste your time budget on likely bad combinations of hyperparameters instead of looking for combinations local to or nearby a known-good combination.

Generally speaking, what GPUs excel at is applying exactly the same operation to many parallel data streams. Neural networks are like that.

Random forests are the opposite: branching on conditional tests is very expensive. It's been several years since I last wrote raw CUDA and OpenCL, but if memory serves, back then the docs essentially said that every if...else amounted to running both branches on all the data and then deciding what to keep, effectively halving performance. So a decision tree just a few levels deep would slow you down by an order of magnitude.

there is a link to a white paper on the github. they show there is a x10 or more decrease in train time on a couple of datasets.

Reply on news.ycombinator.com