Most of the size comes from the LLVM backend, which is a bit heavy. Wasmer ships many backends by default, and if you were to use Wasmer headless that would be just a bit less than a Mb.
If you want, you can always customize the build with only the backends that you are interested in using.
Note: I've seen some builds of LLVM under 5-10Mb, but those require heavy customization. Is clear that we have still some work to do to reduce size on the general build!