I think people end up overselling the difficulty here, because people often strike me as speaking as if the 10ms maximum GC delay is actually the minimum or something and as if the GC is running uncontrollably keyed by a random number generator or something, rather than the amount of garbage generated. It's not that hard to imagine an audio application that runs a 50us GC every several minutes or even less often if you preallocate everything. In practice it's not entirely dissimilar to writing in C and avoiding malloc.
But the real level of difficulty is still something that should make you think, especially as you move from "a single filter" to "a full synthesizer suite".
Rust would generally be a better language, as long as you can step up to the somewhat-harder language, and can find the libraries you need. The payoff for the somewhat-harder language is that you'll have much better assurances about many of the relevant issues. It's what I would generally choose for this task, again, given libraries. But if someone did want to use Go because it's a bit simpler of a language, I wouldn't immediately breath flames on them, unless they were being obviously too ambitious.