This isn't so much Java vs. Go as it is JIT/interpreted vs. AOT-compiled. The numbers are entirely typical across a wide range of such comparisons.
With an interpreter or JIT, you need to load all your code at startup and process it. Generally, _all_ of your dependencies need to be loaded and parsed upfront, either converted to some internal in-memory representation or JIT'd directly to machine code. This will allocate a bunch of heap structures during the processing, and the end result is a bunch of data that needs to stay resident and can't easily be shared with other processes.
With AOT, you mmap() in a file. The OS only pages in the code you execute, and can page it back out as needed. Pages are shared between all processes running the same executable.
At sandstorm.io our rule of thumb is that an app written in Node, Ruby, Python, PHP, etc. will take 100MB of RAM while an app written in C++, Rust, or Go will take 2MB. Since Sandstorm runs per-user (and even per-document) app instances, this is a pretty big deal.
The good news is that https://github.com/google/snappy-start should fix this problem: by checkpointing the process after it finishes its parsing/JITing but before it starts handling requests, we can get an mmap-able starting state that is very much like an AOT-compiled binary. At least, in theory -- there's still a bunch of work to do for this to actually work in practice.
> The resulting docker image shrunk from 668MB to 4.3MB
While I would expect the Go image to be smaller (since Go builds static binaries, so literally all you need in the image is the binary), I suspect that the 668MB Java image was at least 90% unnecessary garbage that was not actually needed at runtime. Unfortunately the package managers we all use are not optimized for containers; instead they evolved targeting systems with dedicated disks that can easily absorb gigabytes of bloat.
In Debian, for example, every package implicitly depends on coreutils. That's perfectly reasonable when installing an OS on a machine: you almost certainly need a working shell to boot and administer your machine. But a container can get by just fine without coreutils, and a typical web server probably (hopefully) doesn't need to call out to a shell. Even if a shell is needed, busybox/toybox is probably sufficient and will take a lot less space.
Packages also often contain things like documentation, unit tests, etc. which obviously aren't needed in a container.
For Sandstorm.io we deal with this problem by running the app in a mode where we trace all the files it actually uses, and then we build a package containing only those. It mostly works and manages to keep packages reasonably-sized, but it does lead to bugs of the form: "I forgot to test this feature while tracing, so the assets it requires didn't make it into the package." We're looking for better options.