Hacker News Reader

Comment by minimaxir | Hacker News Reader

If you haven't kept up with Spark, and do not have Amazon-level workloads, the built-in machine learning APIs are extremely robust and well-documented (https://people.apache.org/~pwendell/spark-nightly/spark-mast...), even in the non-Scala languages like Python. There's even a Multilevel Perceptron model creation function, which creates artificial neural networks using the feed-forward/backpropagation everyone loves.

The new Spark DataFrames also make manipulating data almost as easy as with Python Pandas / R dplyr. The new Berkeley eDX course (https://courses.edx.org/courses/course-v1:BerkeleyX+CS105x+1...) is a very good explainer.

After Spark 2.0.0 is released, it wouldn't surprise me if it really takes off. (as long as setting up a cluster is a bit easier!)

View on HN

> as long as setting up a cluster is a bit easier!

If you're on AWS, it's already quite easy to set up a cluster today, no?

There's EMR of course, and there are tools like spark-ec2 [0] and Flintrock [1].

There are a few more tools listed on Spark Packages that target different cloud providers [2], too.

Disclaimer: I am the primary author of Flintrock and am a contributor to spark-ec2.

[0] https://github.com/amplab/spark-ec2

[1] https://github.com/nchammas/flintrock

[2] https://spark-packages.org/?q=tags%3Adeployment

Setting up Spark clusters is easy relative to setting up clusters, but not easy enough yet to set up and configure relative to simply downloading a package in R/Python.

Well, the absolute easiest way to run Spark is to do it locally (e.g. you can brew install it on a Mac and just go) or to pay for a proprietary service like Databricks, which makes setting up a cluster take a few clicks.

That said, I think `flintrock launch my-cluster` is almost as easy as doing `pip install ...`.

You do need an AWS account and you do need to set your preferences like region and key name in a config file, but I don't see how you can get out of doing even that without subscribing to some managed service like Databricks that abstracts everything away and replaces it with a nice Web UI.

If you're running a DC/OS cluster, it actually is!

> dcos package install spark

This gives you a Spark CLI, and installs the Spark Cluster Dispatcher so you can run jobs async. It also has options for installing the history server, and for SSL/Kerberos support.

Disclaimer: I work at Mesosphere and contribute to the DC/OS Spark package.

Reply on news.ycombinator.com