You might also enjoy ranger[0] which is a console file manager with vi key bindings, I use it as my primary file manager and it's pretty convenient when SSHing as well.
Edit: I should also clarify that if I will work with a remote directory for a non-trivial amount of operations or time, I just mount it locally via sshfs[1] which also saves me the pain of having to install ranger on the remote machine.