You choose a base, and that one is shared across your projects, so the size doesn't matter much within reason. Say, you go for Debian, then you count 50MB download and 100MB space over all your containers / apps, thanks to sharing. So, your stuff goes on top, potentially unshared (say multiple applications).
Which comes to the next contradiction: You then go then for a go-binary, which weighs in by itself easily over a 100MB of binary code. Unshared. Any advantage of self-containment of a go-binary (being statically linked) becomes more a liability, as it is the functionality you gain from docker-images (self-containment), without the sharing of the layers.
If you are focusing on delivering it to some random people, which would maybe care about having to install 100MB on top of whatever you are delivering, then I could see your point, but then you could just go for distributing the go-binaries themselves.
Hah, I'd love something that small. Right now, we have dozens of Node.JS apps, each one needing over 500MBs of NPM libraries..
In the talk, I call this "FROM scratch -- for the rest of us."
That a half-truth because this also tries to set up a minimal compatible environment for common cases, e.g. it sets up ca-certs, a /tmp directory, and an /etc/passwd file.