Hacker News Reader

Comment by bunderbunder | Hacker News Reader

FWIW, you could do the same thing with a Spark cluster. It might not handle it quite as gracefully, since you'd be taking down entire compute nodes, and probably some of the data storage along with them, but the computation would also be able to continue.

View on HN

Yes, but I'm guessing this predates Spark by decades. Not sure of the exact timeframe, though.

In any case, it helps explain why a purse-string-holding exec would be convinced to shell out money to IBM.

spark clusters still have a spof as far as I understand the architecture.

Two of them, I believe - the master for the cluster as a whole, and, for any individual job, its driver.

I wasn't really meaning to say that a Spark cluster is as robust as a mainframe (though maybe someone's figured out some tricks), more that I could totally see some sales folks conducting the same demo using something like Spark.

If you're using Spark's built-in scheduler then the cluster manager is a SPOF. Hadoop docs say you can get active/standy ResourceManager, not that I've tried it. Spark can also use k8s and nomad to schedule executors, and those have HA modes as well. I assume Mesos does HA.

You're still boned if the driver dies. I am pretty sure that the driver keeps some important state in RAM so if the node hosting it goes down you have to restart from the beginning, even if the cluster manager restarts the driver.

Reply on news.ycombinator.com