>Spark takes a Scala AST tree and computes a plan on it in order to distribute it to a cluster of possibly thousands of nodes
Unless Spark has changed dramatically in the year since I used it, that's not really how it works. You lazily construct an expression graph of RDD operators but the actual Scala code (the functional payload of e.g. flatMap) doesn't get analyzed. Are you talking specifically about Spark SQL's code generator?
>The cluster compute benchmarks on it are crazy good.
...and also carefully chosen to show off the niche cases where Spark shines. More commonly, you'll encounter crippling garbage collector stalls leading to mysterious and undebuggable job failures.
Spark has a very clean API but the implementation (at least for Spark versions 0.8-1.0) is still prototype quality.