Compared to a lot of other code, where you can easily tell you're "done" because "it works", with crypto code you're only halfways there. Not only does it have to work, but it has to not break and not leak secrets, too. Anything from timing attacks to bad handling of padding, bad random generators, not to speak of buffer overflows and logic errors (goto fail, anyone?)
I'm thinking it would be prudent to at least use separate keys for anything interfacing with non-default implementations. Can't remember the details but wasn't there an issue where if any (gpg? ssl?) key had been used for signing on a certain flawed implementation, its secrets were spilled?
What's the state of the Go SSH library, has it been vetted by ... veterans? :)