Comparison of machine learning libraries used for classification
github.com
github.com
Vowpal Wabbit does pretty well, which isn't surprising - it has always benchmarked well.
I'm not really familiar with H2O at all, but those are some pretty impressive results. Not only is it competitive in terms of speed with Vowpal Wabbit, bur it also looks like it achieved the highest absolute score too (AUC = 81.2 on a GBM using H2O-3).
~~~~
I was trying to answer each question, but got blocked, so here are some answers in one comment:
1. Yes, it's WIP.
2. Did RF+GBM so far, did not even start DL, but stay tuned...
3. For Spark I used number of partitions = number of cores
4. For why Spark is less accurate for RF see Databricks comments here: http://datascience.la/benchmarking-random-forest-implementat... (essentially current version aggregates votes, while it should aggregate probabilities).
5. The issue of Spark being slow is not only an overhead on small data, but lots of serializing/deserializing, garbage collection etc.
6. Non-linear SVMs scale badly, but one could look at what max size can e.g. svmlight do (results/pull requests are welcome)
7. I think Logistic regression in Python scikit-learn can be much improved (as I mentioned on github) by using a sparse format (someone should do it)
8. The difference between implementations in somewhat surprising, but some of the tools take different approaches (e.g. RF in Python/xgboost works with the data, in H2O/Spark it builds bins from the data etc.) There are tricks, options available only in 1 or 2 etc. I was trying to do my best to match parameter values/setup, but it's not the same.
I also wrote a blog post (a slightly more organized text for RF than the github README): http://datascience.la/benchmarking-random-forest-implementat...
There is also a recording of a talk I gave at a Machine Learning meetup if you want more insights: https://www.youtube.com/watch?v=DK87lCLH_6A
In these experiments, p is quite small(About 1K). So the sizeof the weight vector (or weight gradient vector) is only about 4K (for float) or 8K (for double). It should take less than 0.1 second to transfer these vectors. So there are no need to use tree allreduce or bittorrent to transfer them. These big inventions from VW and Spark become useless and burdensome. And also, in such scene, Spark's accumlator has no advantage than the tranditional map-reduce.
Another point is: Though correctness is more important than performance, it's very difficult to get a correct implementation even for the most basic problems. e.g. Most implementations of OWLQN(which is for L1 regularized convex problems) are wrong. More serious, sometimes the system you relied on is problematic or unstable by design. e.g. This Spark issue https://issues.apache.org/jira/browse/SPARK-5490 may never be fixed.
I would also recommand google's sensei(https://github.com/google/sensei) as an alternative to be evaluated. I recommand it just because it comes from google. But it's not well-known yet.
However - a couple of points.
- One data set can't be more than an indicative test. - Spark isn't designed for single machine environments. - Spark isn't designed for "small problems" ie. where it takes 30 seconds to run the system. - If the algorithms are right then the results for accuracy should be identical. That they aren't indicates a bug, which would be worth unpicking with the relevant teams?
Also, sometimes more data doesn't help much so this is somewhat relevant even for spark users.
In addition, some of the algorithms are non-deterministic. Random forests, as the name implies, involves randomly setting the decision tree parameters numerous times. You can run the same algorithm with the same implementation and get different results. Likewise with NN's, it all depends how you set the weights initially, which is usually a non-deterministic process as well.
To sum it up, I don't think that obtaining different results from different implementations of the same ML algorithm is indicative of a bug. It's actually expected in many situations.
So...
If the variance of a particular implementation from run to run is significant then one has to question how real any gain vs the bottom result obtained is? In the old days we used to do things like cross validate 30+ times and give results x vs y vs z with some confidence interval.
I believe that any implementation of a particular algorithm should be almost exactly the same as any other implementation if it is correct. Therefore, over a number of runs the results should be the same.
However, I wouldn't consider differences in each implementation as a measure of variance. This is likely the result of differences in parameter choices or subtleties in the algorithms. There are multiple ways to train a NN, for example. So I doubt any of the examples shown in the study are true apples-to-apples comparisons.
Some examples of spots where interpretations differ are:
1) Do you exhaustively or randomly search for categorical splits, require one hot encoding or use some other heuristic?
2) Do you use 32 or 64 bit floats? Or do you support both but stop tree growth but have a minimum impurity decrease constant to make sure tests pass on both (scikit does this, it is essentially an unexposed hyper parameter).
3) How do you handle features that have become constant in the branch of a tree being extended? Scikit insists that at least one non constant feature be examined for each potential split resulting in bigger trees, other implementations will examine only m features even if they are all constant. I made this switchable in mine, scikit behavior works better on many test data sets, the other way on some noisy overfitting prone datasets.
4) If default paramaters are used do you default to gini impurity or entropy? Do you round to the nearest number or just up (so you always get at least one) after taking the square root of the number of features to set the number of features examined for each split.
All of this just means it is important to try different implementations and do hyperparameter searches etc on the data you actually care about.
When the dataset size is in Gbs, Non-lineal kernels might not even finish. Kaggle has good sized datasets that you could use to see the difference, it is insane. That said, I would love to see it included in this kind of benchmark to see how ACU would compare.
Seems like mainly he is just saying "DL takes a shitload of resources" and no numbers.
My understanding is that the other tools have limited relevance now with DL, since most systems are moving to cloud-based services that do have resources for DL, and DL offers much better accuracy. That is not readily concluded from this report.
DLs have a shitload of hyper-parameters (effectively, the entire architecture) which is mostly built by trial and error + intuition. There was some really exciting work on reversible SGD by an applied math group at Harvard (https://github.com/HIPS/hypergrad) to obtain derivatives with respect to network architecture, but it's not very mature yet.
Further, DLs need an immense amount of data to be efficient. If your dataset has only a few hundred of features, and you have a few thousand samples, it's unlikely DL will help you much.
Finally, DL is not really useful out of the box, at all. scikit-learn's selling point is that it has an amazingly good uniform API, fantastic documentation, and very clean code. Making a working DL implementation with a simple 'fit' sklearn-like API is impossible.
Conclusions: ...