- you are listening to an SSH session between devices
- and you know what protocol is being talked over the connection (i.e. what they are talking about)
- and the protocol is reasonably predictable
then you gain enough information about the plaintext to start extracting information about the cipher and keys.
It's a non-trivial attack by all means but it's totally feasible. Especially if there's some amount of observable state about the participants being leaked by a third party source (i.e. other services hosted by the participants involved in the same protocol).
So for a contrived example: Say I know a tunnel is transferring a sizeable dataset starting at a specific time before performing some other tasks (say a data sync before doing XYZ). I know when the connection started and I have snooped on the entire connection.
I know the initial handshake and I know the exact plaintext being transferred. That's a lot of information that can be used to grind the keys being used. That then risks that you can extract whatever information that follows after your initial dataset and potentially impersonate a participant and inject your own messages.
It's unlikely to be exploited in practice because it requires a very particular set of circumstances but it's essentially a modern, more expensive version of the attacks used on the enigma machines back in the day. It's unlikely to be exploited on random people but it isn't out of the realm of possibilities for targeted attacks on particularly juicy adversaries or between nation state actors.
You're guessing a cipher key by guessing typed characters with the only information being number of packets sent and the time they were sent at. Good luck. :)