Spoofing TCP is useless - you actually need to receive all the packets sent to the original TCP, which means either you are already on the receiving path, or managed to put yourself on it e.g. through a BGP route advertisement - either way, it leaves some trail and much harder to carry out.
(And even so, the attacker still has to go through SSH authentication or an SSH vulnerability)