Spark 2.0 Technical Preview

tl;dr is yes (to your last question).

The longer answer is that this is about how to logically think about the semantics of computation using a declarative API, and the actual physical execution (e.g. incrementalization, record at a time processing, batching) is then handled by the optimizer.

Neural nets are fantastic, but not all problems either require or are best solved by neural nets. Sometimes you want to understand more deeply how certain variables contribute to the prediction. That's not as trivial to do with neural nets, but in a simple regression you can just look at the coefficients and have some idea of what matters most (assuming you've normalized everything properly).

This is just one advantage of the more traditional models that Spark includes out of the box. Just because neural nets are awesome doesn't mean there aren't big advantages to the older stuff. Also, often it's not the algorithm that matters, it's how you prepare the data and how much data you have in hand. After all is said and done, in many cases the fancy model has similar performance to the boring model.

Very happy to see Spark developing yet more abilities to make ML on large datasets seamless and (relatively) painless.

Maybe there is a lot of talk about it but "everyone" is using Hadoop and the like.

Honest question: what does "in memory" mean exactly? Do traditional databases not process data using memory? Doesn't Spark spill to disk when it reached the limit of the system memory? The in-memory thing always puzzled me.

There are multiple (advanced) projects to get the next 10-100X of perf beyond this via in-memory & streaming GPU computing over columnar data. And... they're taking RDDs in as native formats, and some even target sql as the scripting lang :)

FWIW, my interest here is what becomes possible for sub-second computations. We do interactive analytics, and GPUs become exciting for that, including in the distributed case :) Similar story for stuff like VR, real-time ML, etc.

It often makes sense to just use spark for event processing to create a shaker set of features that are used as input to more robust ML libraries like scikit learn.

Pretty interesting to see watt for watt comparisons : ARM-64 with 64 cores vs GPUs.

Also this is attractive to me because recursion.

I think, as experiments published in last year based on Spark, the inefficiency stems from the assumption that CPU cycles are abundant w.r.t. RAM and bandwidth. So, nobody focuses on optimizing program itself as much as to reduce memory or bandwidth usage.

Some of the performance gain was coming from the use of Unsafe in earlier versions of Spark (e.g. Spark 1.5). However, the massive gain you are seeing in Spark 2.0 are not coming from Unsafe. It is coming from this idea we call "whole-stage code generation", which eliminates virtual function calls and puts intermediate data in CPU registers as much as possible (versus L1/L2/L3 cache or memory).

We will be writing a deep dive blog post about this in the next week or two to talk more about this idea.

In theory, Spark lets you seamlessly write parallel computations without sacrificing expressivity. You perform collections-oriented operations (e.g. flatMap, groupBy) and the computation gets magically distributed across a cluster (alongside all necessary data movement and failure recovery).

In practice, Spark seems to perform reasonably well on smaller in-memory datasets and on some larger benchmarks under the control of Databricks. My experience has been pretty rough for legitimately large datasets (can't fit in RAM across a cluster) -- mysterious failures abound (often related to serialization, fat in-memory representations, and the JVM heap).

The project has been slowly moving toward an improved architecture for working with larger datasets (see Tungsten and DataFrames), so hopefully this new release will actually deliver on the promise of Spark's simple API.

Here's a (simple) problem I solved with Spark:

I had hundreds of gigabytes of JSON logs with many variations in the schema and a lot of noise that had to be cleaned. There were also some joins and filtering that had to be done between each datapoint and an external dataset.

The data does not fit in memory, so you would need to write some special-purpose code to parse this data, clean it, do the join, without making your app crash.

Spark makes this straightforward (especially with its DataFrame API): you just point to the folder where your files are (or an AWS/HDFS/... URI) and write a couple of lines to define the chain of operations you want to do and save the result in a file or just display it. Spark will then run these operations in parallel by splitting the data, processing it and then joining it back (simplifying).

I don't know about videos but I used spark in my last job to solve problems of "we want to run this linear algebra calculation on x00000 user profiles and have it not take forever". For me the big selling point is it lets you write code that can be read as ordinary Scala but which runs on a cluster. As much as anything else it's practical to get the statistician to review the code and say "yes, that is implementing the calculation I asked you to implement" in a way that wouldn't be practical with more "manual" approaches to running calculations in parallel on a cluster.

There are nightly snapshots. Give those a try!