Webpack does way more than esbuild, including running a typechecking compiler instead of just transpiling, running compilers able to downlevel emit to ES5 and providing a deep plugin architecture allowing you to hook into any bit you like. But yes, it hasn't been designed with speed in mind - it has been designed for maximum extensibility instead. Its the same reason why Babel is slow compared to sucrase (written in JS, currently faster than SWC and esbuild but doing somewhat less -
https://github.com/alangpierce/sucrase)
tsc has in fact been designed with speed in mind (I've been following the project since before it moved to GitHub, which is when they redesigned it for speed). Going beyond 1 order of magnitude performance improvement while implementing the full feature set is highly unlikely.