The problem with scp is that the trust also needs to go the other way. There is a lot of ways ssh server can trick the client into doing bad things on your local machine.