I get what you're saying - as someone who is very concerned about bundle sizes currently - but if you are using something like Elm door a help world, you are already losing. Elm is for complex, highly dynamic user interfaces that need a high level of durability and maintainability. Toy projects like the one you describe are useful for learning the language and it's patterns, but for practical purposes it's like bringing in a concrete truck to patch a hole in your driveway.
Also, that bundle size is still smaller than react + react Dom. And you get the features of redux and immutable for free plus a solid, statically checked type system.