Large-Scale Machine Learning with Spark on Amazon EMR | Hacker News Reader

Large-Scale Machine Learning with Spark on Amazon EMR

blogs.aws.amazon.com

47 pointseaxitect11 comments

I do the same on Google Compute Engine, except without the auto-terminate and scaling :(

However, Google's bdutil has a great set of shell scripts which auto setup the environment; and, with minimal changes you can set up the exact Scala/Spark versions you need.

The fact that I (just one dude) can set up a pipeline and chomp through TBs of data on clusters with TBs of memory over the course of hours still keeps me in awe of the advances of both GCE and AWS.

I'll have to give EMR/AWS a shot!

I have nothing against GCE. There's certainly innovation going on there. The new dataflow system certainly includes some very exciting and powerful ideas: https://cloud.google.com/dataflow/

The empowerment of these platforms is something that I'm very excited about. Spark will allow you to go from some fairly basic processing of small files on your laptop to a massive cluster that will process huge amounts of data very efficiently. And EMR makes all of that even easier.

This is something that I hope to convey in my upcoming book: http://www.reactivemachinelearning.com/ One of the ideas that I'm playing with is that big data and small data are basically the same. You should assume that you have an infinite amount of data, and then you'll build your system to handle whatever comes at it. Even if you end up not having a ton of data, you won't be sorry you used awesome tools like Spark.

Jeff here. Glad that people are interested in this post. Feel free to ping me with any questions.

Hi Jeff, congrats on your work. One question of general interest for buddying big data scientists and engineers: do you think from your position that Spark is going to replace Hadoop in the coming future or they will occupy different niches in the market? Thanks.

I use Spark in conjunction with many Hadoop ecosystem mainstays: YARN, HDFS, etc. Hadoop mapreduce can be swapped out for Spark, but many great things beyond that have stemmed from the Hadoop project.

Yep, our Spark deployment, like many others, uses YARN and HDFS. EMR has done some great work to make YARN a great target for deployment of jobs using various technologies.

I'm very much not against the Hadoop ecosystem. The ecosystem represents very real progress for data infrastructure. But Hadoop MapReduce is just not what people should be using to build machine learning jobs at scale in 2015.

Spark makes great use of the Hadoop ecosystem, and I'm primarily interested in future innovations in the big data space that try to work with the Hadoop ecosystem instead of trying to supplant it. Modularity and composability benefit us all.

I think that the Hadoop ecosystem has just expanded. Spark is now a key part of that ecosystem.

Hadoop's MapReduce implementation is arguably quickly becoming obsolete with so many more powerful platforms beating it on several dimensions (not just Spark).

That said, if your interest is machine learning specifically, I can't sell Spark hard enough. Functional programming is so critical for modern, large-scale machine learning. Spark is an absolute revelation for the machine learning developer. It's not the only stack; the PyData stack is totally worthy of study and use. But for big data machine learning, Spark is as perfect as anything has ever been.

If you're really interested in learning more about how Spark, Scala, and functional programming come together in a machine learning system, I'm writing a book on reactive machine learning: http://www.reactivemachinelearning.com/ In it, I'm trying to cover the how and why of different tools, with a focus on Scala, Spark, and Akka.

>By being able to construct on-demand clusters programmatically that auto-terminate on completion, we’ve been able to use ephemeral clusters for all our data jobs. For much of the day, we can have very few data processing clusters running at any given time. But periodically, we spin up many large clusters via EMR that train all of the models that we need to learn. This usage pattern is neither harder to implement nor more expensive than a serial execution of all of our jobs and matches our preferred workflow much better. For our usage pattern, this actually represents a large cost savings over a persistent cluster.

I recently started working at a startup with a similar approach. My previous company had a persistent cluster in a colo, and I found that to be much easier to develop against and use. It takes a significant amount of time to spin up a cluster, which slows down the feedback loop, and makes the development process longer.

I also wonder about cost. It seems to me that if you have enough jobs running every day, at some point you should be able to have them scheduled such that a persistent cluster has fairly high utilization, such that it becomes cheaper than spinning up ad hoc clusters. Have you done any analysis to show that ephemeral clusters are cheaper? Is there a cutoff point where that is no longer true?

I agree that the spin up time is not 0, but how much that matters is really influenced by the size of your jobs.

I would also call out that, when developing against Spark, you can and should be running your jobs locally to develop and debug them. So, I would lean on local Spark execution to keep a tight feedback loop. This is one of the big advantages of using Spark over Hadoop or similar more complex systems. Your jobs scale predictably from a local laptop to a cluster.

Sure, there's a cutoff point, but within a range, this is largely a choice. We're a startup, so our traffic is always increasing. We want to worry about sizing our infrastructure as new jobs come on line. There is no perfect static size for our clusters. We've also been scaling down as well as up in the size of our infrastructure as we improve our code's performance and understand our systems better. So ephemeral just makes sense to us for the moment. Certainly we might want a persistent cluster at some point, but dynamically-scaled out clusters will likely still be valuable at that point.

Thanks for the response. I should note that my experience is with Hadoop, not Spark, which is part of why I was interested in your article.

Sure. The story is somewhat different with Hadoop. But it's really quite feasible to adapt your systems incrementally from Hadoop to Spark. One of the details that I elided in the post is that we're just now decommissioning our last Hadoop machine learning job, nearly a year after moving to Spark. These things take time. That's why it's important that new and useful tools like Spark, Impala, etc. work with the existing Hadoop ecosystem.

Reply on news.ycombinator.com