Excellent write up! Running a single long running Spark Context that accepts and runs jars compiled from user input is a curse I thought we were suffering alone at my company. We too have fought the ContextCleaner, though for us it was G1GC and `-XX:+ExplicitGCInvokesConcurrent` on the driver that saved us.
Is this dynamic class execution a common use case? How frequently are you broadcasting new bytecode to your workers?