Tuning Java Garbage Collection for Spark Applications

databricks.com

The more help the better on this one. I've spent the last week trying to debug why my job fails with a large dataset after 2 hours. I could see that the heap was being exhausted but it took me ages to realise that coalescing to a single output partition meant the entire dataset needed to fit into RAM :-(

I've got to say I enjoy using spark & scala far more than crunch & java.

By the way - does anyone know how to connect to the spark UI when running under YARN on EMR?

The YARN UI has the same information as the Spark UI.

But have to agree that Spark is amazing.

The only thing it's missing that I really need is secondary sorting... I wish they'd add that soon. I've read it can be done but it doesn't look straightforward.

Reply on news.ycombinator.com