This seems like a fairly useful technique, provided the given bytecode file isn't terribly big (since you're uh...converting it into string representation), and that the bytecode isn't some run-once-throw-away code, otherwise you spend more time converting it into Lua code and JIT-ing it.
Having said that, I wonder if it'd be even less of a hassle by skipping the bytecode part. Just transpile the source written in your custom language straight into Lua code, and have that code JIT'ed instead. Many high level structures/information would be preserved, allowing JIT to do an even better job perhaps?