I was really looking forward to using Datasets. unfortunately, I've had much better results sticking with RDDs.

I migrated to 2.0 a few months ago and have been banging my head ever since. RDDs are very straight forward. You get a distributed collection, and you apply functions to them. It's (almost) very explicit about what's going on.

I've been trying to do similar things with Datasets that are super simple with RDDs, but the Dataset / SQL interface hides too many details. I really tried to give them a fair shot. But, I ended up falling back to the RDD interface.

However, in my case, I'm dealing with 50+ TiB of data, so understanding how memory and processing are being used is very important. It's probably less important for casual users.