The new programming model of spark is around Graphframes...which are a Pandas/Dataframes-esque mental model.
Much nicer IMHO
https://databricks.com/blog/2016/03/03/introducing-graphfram...
Much nicer IMHO
https://databricks.com/blog/2016/03/03/introducing-graphfram...
I migrated to 2.0 a few months ago and have been banging my head ever since. RDDs are very straight forward. You get a distributed collection, and you apply functions to them. It's (almost) very explicit about what's going on.
I've been trying to do similar things with Datasets that are super simple with RDDs, but the Dataset / SQL interface hides too many details. I really tried to give them a fair shot. But, I ended up falling back to the RDD interface.
However, in my case, I'm dealing with 50+ TiB of data, so understanding how memory and processing are being used is very important. It's probably less important for casual users.
We're primarily using DataFrames (since we're doing lots of ML processing) and Pipelines API, but I was looking forward to switching to Dataset API.
Do you see yourself getting better performance with the Dataframes+ Pipelines infra...or is it simply a better mental model.
Would you mind shooting me an email rxin at databricks.com so I can understand more the issues you run into?
Can you talk about these limitations and your experience?
https://issues.apache.org/jira/browse/SPARK-1476
https://issues.apache.org/jira/browse/SPARK-6235
You'll hit this bug when your model size is larger than 2GB.
BTW, Recomputation of RDDs may result in duplicated accumulator updates. So please do not use accumulator in your trainer for gradient summation. They know that, but they said they will not fix that.
That is, I see this as akin to linked lists. Something that people typically do not do on their own anymore, but those that do typically do so for a very good reason.