This might be true, but Babel is a big, important project. I’d have expected performance to be something they’ve invested in. That may be wrong.
I think Babel 7 had some perf improvements (including tweaks contributed by Benedikt Meurer from the v8 team), but I don't think they've ever had time to focus on just improving perf.
It's true, speed is not really a concern on my mind at the moment - rather that we have enough people even willing to contribute back at all