I think the main thing is it starts to become a distraction from just writing the gameplay code. I don't have to implement the pooling stuff now that I have this compiler--naive / simple code tends to also start off with a high perf ceiling. But yeah if I did go further with the game in vanilla Go I might have to try the pool approach. Having worked on game engines with GC language runtimes (using Lua etc.) before, you always ultimately hit a perf ceiling due to lack of memory control and wish you could move out of it, but the runtimes don't give you a way to do that incrementally.
Ultimately in the game scenario the GC is actually just ... not helpful. Game logic code already explicitly handles lifetimes to some degree (eg. when this entity collides with that one, destroy it, etc.) -- emergently deciding when to free things based on references is usually not what you want. You do want it for resource management (like a texture cache), but it actually makes sense to kind of roll that on your own and adapt it to the game. So having a GC and then fighting it just sounds like an ill-fitted solution.