You can forward ssh through ssm, and dump that into your ssh config file. Works pretty nicely with some of the sso automation for the cli that's around these days.
Is this using SSM’s raw port forwarding support? From what I’ve seen, their protocols seem to lack binary safety (we get weird encoding issues).
Without knowing the specifics of your situation, I would slightly suspect client/configuration if you’re encountering encoding issues. In my experience, integrating ssh and ssm is quite stable (provided you’re using OpenSSH and not a specific language client’s own implementation of the protocol).