Flexibility usually interacts poorly with performance as well. Specialisation to a domain is usually faster and can be dramatically faster, but the plumbing that makes the specialisation possible has a cost everywhere.
I'm curious whether lisp's macros are sufficient to splice together dissimilar code and provide the domain specific optimisations one would want to eliminate the plumbing overhead. I would expect not but it'll be interesting to be proven wrong.
It's interesting that you're starting with the user interface instead of the sufficiently smart compiler. "I'll rebuild the world in lisp" projects sometimes (usually?) stall in the first stage of writing their compiler. I think it's quite encouraging that you're building on an existing language implementation.
That would be a lot to build in five years. Good luck and happy hacking.