This makes me wonder about newer terminal emulators on maccOS like Warp[1], and if they're for example taking all input locally, and then sending it over the remote host in a single blob or not? I imagine doing so would possibly break any sort of raw-mode input being done on remote host but I'd also imagine that is a detectable situation in which you could switch into a raw keystroke feed as well.
[1]: https://warp.dev