Memory management for a one-and-done image conversion isn't much of an issue. The program will pretty much allocate, run a few
very fast GCs since almost all memory usage is in a handful of big arrays (some people seem to have this idea that all GCs are multi-millisecond ordeals, which isn't the case), and then at program termination just throw everything away. A program like this could get away with an allocator implementation that allocated normally but whose "free" is a no-op [1].
What would slow something like this down is not using the relevant assembly code to accelerate working with these images. A very quick glance at the convolution package, chosen on the grounds that if anything is ASM it probably is, shows that this really is "pure Go", in the sense that it doesn't use Go's ASM support. The compiler will not optimize that to speak of, certainly there's no autovectorization, so it's going to be significantly slower than an optimized program that does make correct use of vectorized operations.
Though, in theory, this can be cleaned up over time. Go does have the support built-in for fixing that in a reasonably nice way, as long as you don't mind that it has its own assembler dialect.
[1]: Not 100% related, but close enough, the part towards the end: http://www.drdobbs.com/cpp/increasing-compiler-speed-by-over...