A little SSH file copy magic at the command line

"You can authenticate with keys" wasn't the punchline I was expecting but perhaps if I had noticed the domain I may have set my expectations differently.

If you copy a file using this method from host B to host C by issuing an scp command on host A, do the bits go directly from host B to host C? or do they go through host A on their way to host C ?

I suspect the latter is what happens on Windows network fileshares, because it takes significantly longer to perform a file transfer from \\hostB to \\hostC while in \\hostA than just doing it direct from \\hostB or \\hostC.

What happens in linux behind the scenes for such 3-party scp transfers?

Any technology demonstrated to a sufficiently unsophisticated audience is indistinguishable from sufficiently advanced technology, which is indistinguishable from magic.

~~ Receiving system `# nc -l -p $arbitraryPort > /path/to/file `

~~ Sending system `# cat /path/to/file | nc $receivingSystemIP -p $arbitraryPort `

eg: `# nc -l -p 9669 > /home/user/staging/httpd.conf ` `# cat /etc/httpd/httpd.conf | nc 10.50.50.1 -p 9669 `

This method is a useful feather to have in one's cap when used in combination with tar and Pipeviewer `pv`.

I've used this several times doing a backup of a large database and when I don't have enough space locally to hold the copy.

So, if you're using it to load code for net booting, scp is the obvious choice since you want to automate the retrieval of a specific file from a specific place, to a specific destination. And you don't want to have to babysit it every time.

If you're copying some specific files with lots of randomish names from your livingroom entertainment server's Downloads directory to your laptop's Documents directory, sftp is your friend.

One doesn't supersede the other in any way whatsoever. There are so many things wrong with the idea that rsync is more modern.

Isn't rsync older by quite a bit? And since it is lacking encryption, isn't scp the more obvious tool to use?

You can't use rsync the way that you use sftp. Think of it this way:

* Secure direct: scp <-- encrypted

* Secure interactive: sftp <-- encrypted

* Insecure direct: rsync <-- clear text

* Insecure interactive: ftp <-- clear text

Use clear text if the network is absolutely known to be secure (good luck on that in any productive environment) and stick to scp and sftp or equivalent secure protocols.

Toss ftp and rsync to the dogs and pretend you've never heard of them except in dire emergencies where significantly better tools (like scp and sftp) aren't available.

There was a recent CVE about writing to wherever the server specified but that does not necessarily reflect poorly on the protocol.

     tar c folder | ssh server tar x

    rsync --filter=':- .gitignore' ...

Is set on `sshd_config`.

And 'scp -c' will automatically compress and decompress your file, and 'scp -R' will walk a file tree for you.

This isn’t me telling you to implement some scary arcane wire-protocol, either. All an AuthorizedKeysCommand script/binary has to do is spit out a stream of SSH pubkeys considered valid to authenticate a login attempt on a given username. The “implicit default” AuthorizedKeysCommand is `cat “/home/$1/.ssh/authorized_keys”`. Your implementation just needs to do the same thing that that command does: take “pubkeys that mean this user could log into this box” from some source, and spit them at stdout. (Don’t want that user logging in to that box? Don’t write anything to stdout.)

I wrote my own little AuthorizedKeysCommand binary a while back (https://github.com/tsutsu/github-auth3) for using Github’s user-SSH-keys and user-org-membership APIs as such a repository. You tell it a Github org name, and it’ll stream out the username’s equivalent Github user’s SSH pubkeys, if-and-only-if that Github user is a member of the specified Github org. (The code is, like, 20 lines. Read it yourself!)

——

This is also one case where specific ecosystems can really smooth things over, though.

If you’re running in Google Cloud, for example, you don’t need to do any of the above. Instead, just configure “OS Login” (https://cloud.google.com/compute/docs/oslogin/), and then log into the gcloud(1) tool on your workstation as an IAM user that you’ve granted the “OS Login” role. Then it’s as simple as:

SSH: `gcloud compute ssh instanceName`

SCP: `gcloud compute scp ./f.txt instanceName:` (or vice-versa)

On first use, IAM will generate a new SSH keypair for the OS Login user. Your gcloud(1) keystore will grab the private half when it needs it; Compute Engine instances will hit IAM to ask for the public half when anyone tries to log in (probably using an SSH `AuthorizedKeysCommand` stanza; I haven’t looked into it.) The user account (`me_example_com` if your IAM user is `me@example.com`) will be created on the instance if it doesn’t already exist (but you can set up your instance template such that it does already exist, if you like, or set up roaming profiles using Filestore as an NFS host, or whatever.)

You can also log into your OS Login account through regular direct SSH/SCP to the instance’s external IP (this isn’t some custom proprietary version of SSH or anything), but you won’t have access to the IAM-held private key if you’re not going through gcloud(1), so you’d want to configure your IAM user with a custom authorized pubkey first (which your instances will also automatically see.)

s/ssh/scp/