File Transfer with SSH, Tee, and Base64 | Hacker News Reader

File Transfer with SSH, Tee, and Base64

susam.in

48 pointssusam63 comments

This is a problem I used to solve with 'zmodem' which you can install on both ends and use to perform text transfers, solely over the interactive channel.

I think the binaries were named 'zs' and 'zr'. It's been a long time since I needed to do this anywhere...

Based on an offhand comment at https://www.extraputty.com/features/zmodem.html, they were `sz` and `rz`.

Wikipedia (https://en.wikipedia.org/wiki/ZMODEM) says that this came out of the BBS days. Boy, that takes me back.

Zmodem, xmodem, kermit. Don't know which one's better etc.

Last time I used one of these (don't remember which) was when my ISP booted me off cause of not paying bill, but minicom still worked, and gave me a shell. This was around 2005.

zmodem was the most advanced in terms of both efficiency and ability to resume broken transfers ...

IIRC smodem was superior (but maybe not so common): https://en.wikipedia.org/wiki/SMODEM

You're close on the names of the binaries. They're 'sz' and 'rz'.

I used the crap out of them dialing-in to systems back in the day. I had TELNET and FTP access to the Internet out of a dial-in to a shell on a Unix host in the mid 90s (no SLIP, PPP, or SLiRP from the provider at the time. It was great to be able to ftp down files to my home directory then 'sz' them back home.

I wish terminal emulators still supported zmodem natively like some of the better emulators many years ago. It's not too difficult to configure iTerm2 to invoke rz when detecting the escape sequence[1], but it's still too much friction. sz/rz is something I would be using everyday, but I prefer sticking to Terminal.app on macOS--I prefer zero-config environments because of all the various systems, local and remote, I interact with on a daily basis.

scp and sftp are great, but the nice thing about in-band file transfers is that your shell session already has the current context for simplifying the transfer. For example, you often want to send to or receive from the current working directory, or something nearby, and interactive file globbing is just a single tab key away, without having to switch sessions, login all over again, copy paths, etc.

[1] I don't miss automatic printing without confirmation, though. The prank of `cat /vmlinuz | write user` wasn't that funny when the service desk hands you an entire ream of garbage along with a reprimand.

Gnu screen supports zmodem send and receive. I use it frequently to get files into old retro computers I’m playing with.

I’ve used it in datacenters where I’m ssh’d through three intermediate hosts, or when I’m inception-ing down into containers and vm’s

Tweeted about zmodem just last night! https://twitter.com/ryancnelson/status/1306444057602711553

My solution in these cases:

    ssh hostname tar cvjf - /path/to/folder | tar xjf -

Basically I ask ssh to execute tar on the remote host to create a compressed archive. ssh will output the archive contents on the local host; this data is then passed on to a local tar for extraction.

I do mention in my blog post that the SSH gateway in between forbids execution of remote commands without a login shell, so this rules this out as the solution. Otherwise, yes, this would have been a fine solution.

Honestly, I find it distasteful to have to spend time working around somebody's incompetence at securing systems.

Doing it on your time means you delay delivering on your project and you let whoever did this get away with wasting everybody elses time.

> Honestly, I find it distasteful to have to spend time working around somebody's incompetence at securing systems.

> Doing it on your time means you delay delivering on your project and you let whoever did this get away with wasting everybody elses time.

While it may be distasteful, what's the alternative? Refusing on principle to use a system configured in a way you don't like is way more likely to hurt you than it is to hurt anyone else, especially the person who configured the system.

Even if you are doing work for someone else (which, I think, is not indicated in the post), so that that person will be affected by your principled refusal, there's no guarantee that they're the ones who misconfigured the environment in which you're operating.

Your patience in (re-)explaining this constraint to everyone in this thread who thinks you don't know your way around standard Unix tools is impressive. :-)

I went around a very similar issue by doing it in reverse.

Rather than pull via SSH I login to the remote interactive shell and dump the file onto my own server via SSH from that remote shell.

Would this make it through the SSH gateway? It creates a pseudo TTY and executes bash as a login shell within it.

  ssh hostname -qt 'bash -lc "tar czf - /path/to/folder 2>/dev/null | base64"' | base64 -id | tar xvzf -

Probably want to prepend a prefix so you can extract only those lines and filter banners, motd's, etc, like:

  "... base64 | sed -e 's/^/~/'" | sed -ne 's/^~//p' | base64 ...`

BTW, base64 isn't a standard command (nonexistent on OpenBSD), and the options can vary based on implementation even when it exists (decode is -d for GNU coreutils base64, but -D for the macOS command). You can whip up an alternative using od(1) and printf(1):

  #!/bin/sh
  # encode as backslash-escaped octal (\0NNN)
  export LC_ALL=C
  od -An -to1 -v \
  | sed -ne 's/\([01234567][01234567]*\)/\\0\1/gp' \
  | tr -cd '\\01234567\n'

  #!/bin/sh
  # decode backslash-escaped octal
  export LC_ALL=C
  while read -r L; do
    printf "%b" "$L"
  done

It might not be strictly POSIX compliant, but I don't think there are any environments where printf isn't 8-bit clean in the C locale. I've tested something similar (see below) on various versions of bash and d?ash, as well as /bin/sh on AIX, FreeBSD, macOS, OpenBSD, NetBSD, and Solaris.

You can implement base64 as a shell script using the above techniques, and do so without relying on awk, which isn't always available. If od(1) isn't available you can use dd(1) to extract stdin byte-by-byte and convert with printf(1), but that gets really slow. I've written an implementation that only needs an 8-bit-clean but otherwise POSIX compliant shell, printf, and either dd or od. It doesn't need tr, but it's even slower without it. It just converts the bytes to decimal and uses plain shell arithmetic for the primitive encoding and decoding, similar to how it would be done in any language.

It certainly isn't a complete cross platform solution but to address the inconsistent options problem and some of the availability issues openssl could be used instead.

  ... | openssl enc -base64" | openssl -base64 -d | ...

Self reply. Noticed I left out the enc option from the 2nd openssl command. Should be:

  ... | openssl enc -base64" | openssl enc -base64 -d | ...

This seems like a CTF trick. If I were faced with this problem, I would do an SSH local port forward, and then just wget the file, hosted locally with python -m SimpleHTTPServer

Or you can open SSH back to your local port forwarded server from the remote SSH shell. At that point you can pipe whatever you want to/from your local server using the standard SSH commands.

I think we can assume that if they are restricting things as the article describes they also disabled port forwarding.

Port forwarding your local machine is always an option.

That's right. See https://news.ycombinator.com/item?id=24507146 for example.

I mentioned in my blog post that there is an SSH gateway in between. I tried this on my local system:

  ssh -L 9000:127.0.0.1:8000 user@host

But when I visit http://127.0.0.1:9000/ on my local system, I get a connection reset by peer error. At the same time, in the terminal where I have set up the port forward with the ssh command above, I get this error:

  open failed: unknown channel type

The SSH gateway is forbidding port forwarding as well.

I know we're getting into the weeds here but for the benefit of people learning random things that work slightly differently (tm)

A possible workaround for this specific scenario would be to use 'sshutle' which runs a python script on the remote end to terminate the traffic.

Except in your specific scenario you'd have to somehow trick it to execute using a login shell like one of the other comments shows.

Having said all of that, I would just do what you did :)

The instructions in this article should work, but they're more complicated than necessary maybe. If you don't have scp, you can usually just do something like this:

  ssh user@host cat remote_file > local_file

to copy the contents of "remote_file" on the server to "local_file" on your local machine.

> If you don't have scp [...]

scp is deprecated [1]; use sftp instead. At the very least, sftp should be your preference over scp.

[1] https://unix.stackexchange.com/questions/571293/is-scp-unsaf...

Scp works fine, I use it all the time. It's less complicated than sftp, and works basically just like the regular 'cp' command. It also doesn't require an sftp server to be present on the remote machine.

I'm not 100% sure I buy those "it's less secure" arguments in that Stack-exchange post, but even if true - lots of people use SSH to move stuff around inside an intranet, where both ends of the connection are trusted. In that kind of context, scp works great.

Which parts of the facts listed do you not "buy" ?

The fact that a remote server gets to silently respond to the request "Hey, I'd like funny-cat-noises.mp3" by writing two files, funny-cat-noises.mp3 and also .security/overrides.conf ?

Maybe your understanding of the protocol was that you allow a remote server to write whatever files it wants to your disk. That seems like a pretty crazy thing to want, but OK.

Most people assume it does what it seems like it does. You name one or more files on one machine, and they're copied to the other machine. SFTP can deliver without nasty surprises, but SCP can't really do so, although the OpenSSH team have worked pretty hard to make the worst surprises less likely to bite you.

That sounds like it's an issue if one is copying from an untrusted server, for sure.

Maybe not an issue if you trust the machine you're copying to/from.

> Maybe not an issue if you trust the machine you're copying to/from.

Maybe you shouldn't. After all, the remote machine doesn't have root on your machine. Even if you are root on the remote machine, how can you know the server isn't compromised? You can't.

Your link doesn't say it's deprecated...

the sftp command has lousy ux, it's unlikely people will switch to it en masse.

> The scp protocol is outdated, inflexible and not readily fixed. We recommend the use of more modern protocols like sftp and rsync for file transfer instead. [1]

> From The Jargon File (version 4.4.7, 29 Dec 2003) [jargon]:

> deprecated adj.

> Said of a program or feature that is considered obsolescent and in the process of being phased out, usually in favor of a specified replacement. [..]

[1] https://www.openssh.com/txt/release-8.0

But your cited definition agrees with me? It's not being phased out, hence not being deprecated.

> The scp protocol is outdated, inflexible and not readily fixed. We recommend the use of more modern protocols like sftp and rsync for file transfer instead.

I'm not sure what the misunderstanding is here.. The above text says there are better protocols and recommends their use over scp, but doesn't say scp is being phased out or discontinued in the future.

On *ix things stay around for pretty long. You still have ed and ftp on Ubuntu 20.04!

rsync has pretty much identical ux to scp and is actively maintained.

If you can do that can’t you just use rsync?

I have a convoluted method in my blog post due to the constraints I was working with. The SSH connection to the host goes via an SSH gateway that forbids the execution of remote commands without a login shell. I have laid out this constraint in the first paragraph of my blog post.

Without this constraint, yes, I agree what you mention would be the obvious straightforward solution.

Really glad someone mentioned this. Works fine with binary data (doesn't need to be base64 encoded), and is much simpler.

If the ssh gateway OP mentions is just a ssh server, a ProxyCommand should be used, and the gateway should be transparent after that.

    Host rcpu-tun
      User YOURLOGIN
      IdentityFile ~/.ssh/YOURSECRET
      ProxyCommand ssh -q -W HOSTBEHINDFW:22 -l PROXYLOGIN SSHGATEWAY
      HostName HOSTBEHINDFW

ssh -W requires that the gateway support port forwarding, which is probably disabled if its enforcing interactive logins. (-W also implicitly sets -T, though I don't know if that would fail the login or not.) Likewise, -J is shorthand for a ProxyCommand that runs nc on the gateway, which also won't work if the gateway requires interactive (i.e. TTY interposed) logins. (EDIT: Looking at the OpenSSH source code, -J is shorthand for generating a ProxyCommand using ssh -W. Either it was refactored to use -W after -W was added, or I'm misremembering how -J used to work.)

If you can SSH directly into both systems, then you don't need to print anything directly to the terminal. You can go:

  ssh system1 'cat payload' > temp
  cat temp | ssh system2 'cat > payload'

Or cut out the middleman, as long as getting two password prompts isn't too confusing:

  ssh system1 'cat payload' | ssh system2 'cat > payload'

I've found the base64/copy-paste trick useful when I can't do a conventional SSH. Usually this is when one of the machines is accessed via a serial console.

In that scenario, it can help to compress the data before base64-ing it:

  machine1$ cat payload | gzip | base64
  <output>
  machine2$ base64 -d | gzip -d > payload
  <paste output>

They mention that you can’t run commands without an interactive shell. I believe that knocks out your proposal because then rsync+ssh would work without all the complexity you have in your post with all the same benefits. Could be wrong though.

I see, I did miss that.

I do wonder if there's a workaround for that, though. "bash" has the "-l" option for "Make bash act as if it had been invoked as a login shell", and also "-i" forces interactive mode. For ssh, "-t" is "Force pseudo-terminal allocation"; would that work?

The SSH gateway in between refuses to allocate PTY, so the -t option does not help.

I don't follow. How would any of these option help?

> Usually this is when one of the machines is accessed via a serial console.

I've found zssh can be quite handy for that, it layers an extra pty over the SSH connection, for file transfer. By default it uses the ZMODEM protocol via rz/sz, but any pair of programs can be plugged in instead - useful if the connection isn't 8-bit clean.

I just confirmed this can also work using AWS Session Manager (a "secure" and easily-auditable way to provide shell access to EC2 instances):

    $ aws ssm start-session  --target i-XXXXXX | tee base64.txt

I do something similar to move small files via copy-and-paste to/from hosts that I have to reach through an intermediate jump host.

Rather than copying the file to the jump host, then to the remote host, I'll just gzip and base64 encode it, then copy and paste it to the remote host.

I've done this to get files into VMs that have poor or no network connectivity, too. If I can get a console session w/ the ability to paste keystrokes I'm golden... >smile<

Heh, thought I was not alone! Although I do not use SSH, I just use netcat. Shared directories are a hassle. :D

That's a litmus test for me when working with other technicians. If they need to transfer a file to me across a shared LAN and they offer to pipe the file over Netcat (and they just expect that I know what they mean) then they "rate". If I suggest Netcat and they understand they "rate". If they have no idea what I'm talking about or, worse, suggest a "cloud" service, then I file them appropriately.

Edit: Just to be clear, I'm talking about transient situations-- like when I'm working with another contractor on a day-long gig, etc. Obviously, for an established LAN piping files to each other over isn't a sensible or secure file transfer method.

I do not know why you are being down-voted. I would expect people to know what netcat is, and what it is used for. If all someone can come up with is related to cloud, then I would dismiss them, too. These "problems" have been solved a very long time ago, and we have great alternatives.

For myself I have written two shell functions; nc_send and nc_recv. It uses Nmap's ncat with SSL. It uses GPG and to verify data integrity, I use b2sum. Simple, yet perfect for the task. I use it often besides scp, but of course sometimes netcat is all I need.

I wouldn't necessarily "dismiss" somebody, but knowing about simple tools is a proxy in my mind for experience level. I can tailor my communication appropriately when I have some idea of experience level. If I know I can't say to somebody plugged-in to my laptop directly w/ a patch cable "Give me the last two octets of your APIPA address and start listening on port 31338 piped out to a file" then I know that there are a lot of other tasks that I'll need to break down in minute detail.

Okay, you are right. I would not dismiss them either for this alone, I would first perhaps ask them why they think that it is the best option, or what alternatives there are, and so forth.

> Give me the last two octets of your APIPA address

Is not APIPA Windows-only, by the way?

I can definitely give you two octets of an IP address through using awk, sed, or using C and bitwise operations. Do I pass? :D

APIPA was standardized as IPv4LL in RFC 3927[1]. I never remember the acronym IPv4LL-- I always remember APIPA instead. It was a Microsoft construct, but it has fairly broad OS support now, to my knowledge.

[1] https://tools.ietf.org/html/rfc3927

I am trying to transition myself from netcat to socat. It's wordier, but more versatile and complete.

> sed '1,/$ base64/d;/$ exit/,$d' ssh.txt

Huh. I'm surpised this works, because $ is not escaped.

Single quotes.

No, I mean $ usually means end-of-string (or end-of-line) in regexps. But TIL in BREs, $ means literal character in the middle of an expression:

https://pubs.opengroup.org/onlinepubs/007904975/basedefs/xbd...

Oh I see now, I never thought about $ being usable anywhere except at the end as an anchor!

scp is so slow, does anyone want to improve this?

Reply on news.ycombinator.com