Sucrase is faster or really close to SWC (see rhe benchmarks https://github.com/alangpierce/sucrase). Everyone still uses Babel because of the transforms.

And yes, Babel can also be made faster if enough effort is dedicated into it. it's not an impossible feat.