Need to handle non-js assets in your bundle? Need to integrate into both node and browser environments?
You should stick with webpack. Any time you lose waiting for webpack to run you will get back 10x over by not fighting with your tooling because it doesn't handle a set of use cases you have.
I'm sure the next generation of build tools and bundlers will mature over time and we'll all get to enjoy the benefits of es modules, but right now webpack's mature plugin ecosystem, documentation, and stability makes it my default choice.