This is not really true for SPA's, but I hope that's a trend that will eventually sizzle out again except for specialized cases.
Just grabbing what you need can have significant benefits. I agree with the latency problem though, it would be nice to have a HTTP/2 or HTTP/3 server that has a basic understanding of the JS dependency tree and preemptively push the entire tree.
1. Typically, you only ever care about minimizing bundle size in cases where latency is consistently questionable
2. In those cases, whatever UI you're supporting is likely then not meant for latency-sensitive operations, and thus the overall workflow tends to have low state complexity
3. Just how much actual value is added in those scenarios by having the ESM workflow specifically in conjunction with importable NPM modules?
I'm sure you can think up some contrived use cases but I'm just not seeing any particularly compelling ones.
I suspect we'll see the opposite, with people building desktop type applications and deploying through the browser (especially with the growth of wasm)