Exactly: 99% of the time, "webpack is slow" is just code for "I have Babel in my toolchain" and forgetting to set ts-loader as "transpile only".
Personally I removed it years ago, and Webpack 5 even allowed me to get rid of more loaders now that there are Assets Modules that automatically detect assets and webworkers using the "new URL()" syntax, and Typescript does everything else I need.