I've done a similar thing with Scala in the past - ship the compiler jar and then compile your configuration (which is just an abstract class implementation) as part of the startup process.