There are 3 big reasons for large Go binaries.
1. Unicode data is ~1 MB. It's needed to implement all the unicode-aware string functions like strings.EqualFold()
2. reflection. For reflection to work, every struct used in a Go program needs to have a description information that tells names and types of all fields. This adds up
3. Precise garbage collector also requires struct descriptions like reflection but even more information like: layout of stack frame for every function.
Those are problems that could, in theory, be worked-around.
A build could exclude Unicode info and functions that require it.
A different garbage collector could be used that doesn't require knowing layout of structs and stack frames.
Reflection could be changed to require explicit opt-in (99% of structs are not reflected upon).
Realistically none of that will happen in the official Go implementation. For better or worse most of the work is sponsored by Google and Google runs Go on servers. With 512 MB of memory, it makes little difference if your binary is 10 MB vs 9 MB, especially if you compete with even more bloat (in runtime memory use, not necessarily binary size) of Java / Python / Ruby / Node.
There are a couple of not-very-mature Go-like implementation that are trying to address this (tinygo, emgo) but they have long way to go.