IMHO, the speed of the build tool is less of a priority than the speed of the code it produces. You pay the cost of the build tool mostly in integration tests or production builds but shouldn't pay it during development.
However your users pay for inefficiencies times the number of times the app is loaded. The state of JS tooling is woefully behind other languages and IMHO still behind the state of the art 5 years ago. Any non trivial sized app for example will require global dead code analysis and method and field level pruning as your dependencies grow over the lifetime of your app, all apps tend to bloat from unneeded code pulled in.