Interesting.
I built the same thing. Through some basic script examples, such as `while(true) {}` or `new Buffer(1000 * 1000);`, the conclusion that I came to was that it is impossible to truly interrupt/prevent resource exhaustion attacks. A bug anywhere that would allow reflection to be accessible would have been disastrous as well.
In the end, I built a JNI wrapper over Duktape+flatbuffers, which gave me super fine-grained control over memory allocation and CPU time.
I'd be very interested to hear a summary of how/whether you managed to make Nashorn safe for user scripts.