35-year-old vulnerability discovered in scp
sintonen.fi
sintonen.fi
-unsafe allow server-side wildcards (DANGEROUS)
"This is due to a fundamental insecurity in the old-style SCP protocol: the client sends the wildcard string (*.c) to the server, and the server sends back a sequence of file names that match the wildcard pattern. However, there is nothing to stop the server sending back a different pattern and writing over one of your other files"
I haven't used the Linux version much so I assumed it had the same option. PSCP has had this option for as long as I can remember. I guess no one bothered to look at scp, or as one of the other comments here notes, scp is overwhelmingly used with a server one already trusts.
It's the latter for me. It's not just wildcards. You can use any server-side shell code you want to specify the files as if you're writing in a command argument[1]. At least, I find this tremendously useful.
The last time I tried to help someone get it set up on a windows PC, totally normal ssh2 rsa 2048 and 4096 bit public/private key pairs created with openssh had to be converted into some other weird format before they could get public/private key auth working.
Why the developers of putty felt they needed to deviate from standard ssh2-rsa pub/privkey formats is a mystery to me.
Reading and writing OpenSSH-style keys came later (2012).
https://www.chiark.greenend.org.uk/~sgtatham/putty/changes.h...
If you're accepting incorrect host keys, then you're completely breaking the ssh security model anyway.
I guess I don't understand how this is actually any worse than how ssh already worked?
Edit: To clarify, I understand why you’d want to fix this, for a good depth of defense. I’m just saying I don’t feel any urgency in fixing this.
Many of us have jobs in which we typically only use SSH to connect to servers that we trust and control. But that assumption is no more baked into the security model of SSH than it is into the security model of HTTPS. The argument that "you trusted this server enough to connect to it and download a file, therefore you clearly should trust it enough to permit it to execute arbitrary executables on your machine" is false in both cases.
Great point. Also, imagine that you control the server and you know that it was compromised. Surely you want to be able to download logs and other files from it for inspection without having your own machine compromised.
Uhm, nope: you want to shut it down and perform forensics on the disk. Once it's rooted, I wouldn't touch it with a barge pole.
1. Shut down the instance
2. Connect the storage as secondary storage on another (disconnected from the network) instance
3. Do forensics using your cloud provider's out-of-band management interface
4. Throw away that instance as soon as you're done
We trust the browser not to have bugs, the TLS protocol to remain secure against attack, every CA to not grant MITM power to a state actor, the TCP/IP stack not to be remotely exploitable. Much of the world downloads unsigned code and exectues it locally, whether it be a Go client, an interpreted library code dependency, or "curl | bash". Windows has warnings before you run unsigned code, but most people run it anyway.
We trust a lot of things, and maybe we shouldn't. The important thing, to me, is how informed I am about the trust I'm giving, and to whom, and what risks I'm willing to take.
I use scp infrequently and on machines that I control, so that's a level of risk I'm comfortable with. But if the bug had been in curl, my blood pressure might be slightly higher.
Unless you run your browser without an ad blocker
On the one hand, you have a real point. On the other, JavaScript running in my browser has significantly less power to do anything bad to me than arbitrary executable programs running directly in my OS do.
I think that depends on what you mean by "significantly less power". It's entirely possible to use Javascript to place malware (such as installing a binary executable) and do other assorted nastiness on a target machine. If the target machine is properly secured, it's still possible, it just requires more effort.
This is the primary reason why I do not allow Javascript to run on my machines by default. If I'm at a site that I trust and I have no alternative to using, and the JS in question is essential, then I'll allow just the specific piece of JS to run. Otherwise, it's not happening.
This sounds interesting. Can you kindly link an example for this technique?
Your best bet (not guaranteed) is reading the html as plain text, not rendering any images, etc.
Nicely put.
This feels like yet another variant of the same confusion we see around web browsing, with people saying "oh, if you don't want a virus don't go to sketchy sites". Agreeing to receive HTML or even run unfamiliar Javascript in a browser shouldn't be equated with trusting that website to run anything outside the sandbox, and in a world where ads on major sites are frequently resold through a half-dozen shoddy networks it's an important distinction.
Even from a host you use all the time and where know the key hasn't changed?
Or use out-of-band access, e.g. a host-provided console, to connect once and extract the key.
For what it’s worth: The way to solve this is require the host key fingerprint as an argument to the ssh command, along with host and user. This would force people to look it up.
This is not a popular opinion, but I think it’s inescapable: SSH was not made for humans.
(This is purely a client issue, by the way. But at the end of the day: does that matter for anyone?)
(Edit: when this comes up people often mention key based auth. Unfortunately, that relies on nobody else having your public key. You can find mine anywhere, it used to be available through the GitHub api , even. It won’t let the attacker connect to the original server but they can definitely impersonate it. Perhaps get your key agent forwarded, and continue. Not great.)
Asymmetric key is not a solution to the problem of users blindly trusting hosts without checking the fingerprint. It’s a UI problem.
Every time I bring this up it gets shot down (e.g. now -3 votes), and every time I wait for someone to put forward an actual counter argument. Unfortunately, no luck.
SSH is broken and it needs fixing.
See it like this:
The bug is in the verification of the host, not the user. Eve can intercept a connection from Alice to Bob. It can "pretend" to be Bob, but it can't MITM (aka "proxy") because it won't be able to auth with Bob. However, auth depends on the server encrypting a message with Alice's public key, which everyone has ( https://github.com/torvalds.keys !), including Eve, so Alice will connect to Eve. Really this is nothing different from just connecting to a fresh host: you upload your public key to the remote ~/.ssh/authorized_keys somehow, and voila, right? How does your ssh client know that was the same server? The fingerprint. Did you check it? No.
Check out this article:
> The more well-discussed use of asymmetrical encryption with SSH comes from SSH key-based authentication. SSH key pairs can be used to authenticate a client to a server. The client creates a key pair and then uploads the public key to any remote server it wishes to access. This is placed in a file called authorized_keys within the ~/.ssh directory in the user account's home directory on the remote server.
> After the symmetrical encryption is established to secure communications between the server and client, the client must authenticate to be allowed access. The server can use the public key in this file to encrypt a challenge message to the client. If the client can prove that it was able to decrypt this message, it has demonstrated that it owns the associated private key. The server then can set up the environment for the client.
- https://www.digitalocean.com/community/tutorials/understandi...
As you can see, ssh public key auth authenticates the user, not the host. anyone could be that host.
Now: you as a human will quickly notice "where is my homedir? wait a second... this is not Bob—This is Eve!" the question is: quickly enough? If you were using scp: no, look at TFA. If you were connecting to github for the first time using git, perhaps using homebrew on a fresh machine: now you'll definitely never know. If you had set up your ssh key agent to forward keys to your "trusted" Bob host: serious trouble. There are loads of reasons why intercepting an SSH connection is a dramatic bug, even without being able to MITM it.
Or just try and turn it around: why do you think SSH prompts you to confirm the host key fingerprint, if it doesn't matter? For fun? They didn't add that as a joke.
Alas, few people understand this, as you so elegantly demonstrated :/
The host key fingerprint does matter. That's the host's public key. The host also has a private key. Without that private key, Bob can't pretend to be Eve.
You are correct that when connecting to Bob, if his public key doesn't match Eve's expected key, it will prompt the user to cancel the connection.
You argue: "It will only require the public key (which is public) to impersonate the server." The key piece here you are missing is that if Bob steals Eve's public key, he won't have a matching private key, and authentication will fail.
If you don't think the private keys are important, go edit them on your server, and see how SSH'ing in goes for you.
The attack is on Alice, who thinks she's connecting to Bob, but is actually connecting to Eve. This is on initial connect, and assuming Alice doesn't check the host fingerprint (which nobody does; that's the crux of my point).
This nomenclature is the convention when discussing cryptographic attacks: https://en.wikipedia.org/wiki/Alice_and_Bob
I'm going to stop discussing this in this thread, but feel free to contact me (info in profile). I promise to engage, in good faith, and if you do end up convincing me, I'll post an update here.
Best wishes.
You're fundamentally misunderstanding the argument about which key is being copied. It's Alice's public key that's copied. The fake server generates its own public and private keys and nobody notices that it doesn't match the real server. The user won't be "prompted to cancel" when this is their first time connecting.
authorized_hosts, as typically used, prevents an imposter from showing up later. It does absolutely nothing to prevent an imposter that's there from the start.
I've got a script sitting around here somewhere which does this... Not necessarily in a state that's easy to reuse, of course.
I did notice recently that cloud init has an option to hot a web hook when it is complete and they have options in there to add the host key to the POST. But I wonder how to make a web hook that would be immune to spoofing…
If after all that you receive a message an error from SSH that the host key identification changed, you didn't change this key yourself (or your team) and you still blindly ignore this issue (since it is not easy to bypass either, there is no flag, for example), this is not an UI issue and you're responsible for your actions.
I wrote about them some time back: https://ayesh.me/sshfp-verification
For servers without a domain pointed to it, I suppose you can still verify them by comparing the history key fingerprint by logging into said server with the console provided by server vendor.
The attack presumes your home directory is client side, and you use SCP to connect to any kind of server, whilst ~ is the client side open directory.
Thus, any server you might SCP to could write to your local home dir. In university, I did this with compute clusters, servers of my association and other servers.
This breaks the SCP security model because it means a server has covert access to your local working path. Whereas normally you know which files SCP touched, so you can verify they are as intended.
I just don’t understand the use case of scp’ing from an untrusted host.
Throw a Tectia server in the DMZ, fix the firewall, manage some keys (never change them) and you're good to go! Data at rest security via Zip or PGP for the ambitious.
Occasionally a big enough entity will mandate "everyone will use our data xfer spec" (and it's typically some gnarly XML-infested SOAPy animal). But there's a whole lot of automated business processes doing CSV-over-scp both intra- and inter-company.
Don't even get me started on FTP.
In which case, you leave your CSV-over-scp or CSV-over-FTP in place, and duct tape on another layer that handles the new data transfer spec. That way you can leave the 8 year old process itself alone and let it keep creaking away, silently chugging away as it slowly fades into the twilight of "important but forgotten" systems running on autopilot throughout the years.
It reminds me of how places like Rome still have sewers from the days of Ancient Rome in operation. Rather than replace them outright, they were just connected them to the newer sewer system to divert the waste to treatment plants instead of directly to water sources. And they'll keep on going, only being updated when absolutely necessary to warrant it.
Ideally your organization should keep updated (public) host keys somewhere, say on some https-protected website so you can double check yourself. How common is this?
(I mainly use ssh for interactive/tunneling use, but with a bit of bad luck the host key would change just in time for scp. BTW, don't rsync/unison use scp sometimes?)
Haven't yet encountered it in the wild though
We (rsync.net) are doing this in 2019. A little embarrassed we haven't done it already ...
Also, when I reinstall a server and keep the name, I preserve the keys from the original server.
I trust the cluster enough to give me those results. I don't trust it with write access to .bash_aliases . Thats why I did
scp file@host .
scp *@host .You are SCPing to an untrusted machine, not from. Your client is trusted, the server you are connecting to is not.
Sometimes circumstances push you to connect to a server that is "semi-trusted" as described in the comments above.
What? Why would it even be an expectation that the host ought to be trusted? Would you say that about FTP, or HTTP or any other file transfer protocol? What's special about SCP?
And you may not even notice that the web server has been compromised until compromised PCs start acting up.
Should be fixed, but clearly not the worse vulnerability ever.
why you would want to fix this; client needs to verify the input it receives from server and not trust it's the right thing. malicious servers is just like how browsers get owned u know. it's not magic and very risky considering how many people have ssh / scp servers in their internal networks, which is just the kind of environment you can get away with a mitm attack on..
If you exclusively SCP to servers you control, this is admittedly a low priority issue.
However some companies use SCP extensively as a way to exchange data files (as an upgrade from using FTP, and out of frustration from dealing with implicit/explicit FTPS modes). Healthcare and finance come to mind for those use cases.
How did you arrive to the conclusion that the server has your home directory? The threat model: It's a server that you can authenticate to, retrieve files from there but otherwise you don't trust. It has nothing to do with your home directory. The PoC attack just shows the danger of invoking scp from the client's home directory.
That makes sense I suppose. I never scp files from untrusted hosts, nor can I think of a use case to do so, so it hadn’t really crossed my mind.
If I'm reading the OP right, whilst directory permissions can be modified, traversal is not possible, so it can't overwrite outside the directory you're writing to. But if you wrote to your home directly then your .bashrc could be modified ... and it sounds like other vulnerabilities allow the actual processing to be hidden (spoofed console output).
Seems like the sort of thing that could be used in a spear-fishing scenario.
I wonder if you could write a StackExchange answer with instructions to scp files from such-and-such server ...
This is not at all uncommon. Consider I need to send you files, either once or on a batch schedule.
You say, "give me your key, you can drop them on this server"
I feel safe since I am just writing files to your server, not expecting you are going to drop arbitrary code that will run next time I open my terminal.
But if I wanted to share files with you, I’d just put them on the web, unless we already have a relationship where you can ssh somewhere that I have access to.
In which case I assume we’re trusted or you’ve at least verified the host key.
If you "trust" any of these things is irrelevant, as that trust might be misplaced and the security model should keep you safe with compromised servers regardless.
I feel safe, because I think I'm going to cat a text file, not execute it.
Well but what if the server doesn't tell you it's compromised?
or, it means the otherwise previously trusted server has been compromised, and the sshd has been replaced with a malicious one.
From what I understand - if you scp any file from a remote server to your home directory, malicious remote server can copy anything else into your home directory without you knowing it, which can lead to remote code execution.
However, (1) the server needs to be malicious or compromised (or mitmd and you accept the new cert), (2) you need to copy into your home folder, not anywhere else, directory traversal is impossible
The progress meter is meant to be a single line, and the formatting calculations simplistically assume that every character in the buffer is a (single-width) graphic or SPC. Anything in the filename that breaks that will disrupt the progress meter.
* [...] only directory traversal attacks are prevented
* [...] can overwrite arbitrary files in the scp client target directory
* [...] the server can manipulate subdirectories as well
... so nothing points to the ability of the server to "fiddle with" parent directories.
Thus.. Is it an OK temporary workaround to _only_ perform scp from within a freshly created directory in /tmp/?
> Malicious scp server can write arbitrary files to scp target directory
looks like you don't need to `cd /tmp` before running scp, it's enough to scp _to_ a freshly created directory (`scp host:file /tmp/dir1`), inspect its contents and `rm -rf /tmp/dir1` afterwards.
What makes MitMs possible is that checking fingerprint of new host is left to user. Instead we should have processes to automatically 1. acquire via other means and add fingerprint of each new target host 2. verify new host fingerprints with a separate party (central server or a ring of trusted buddies).
All you need to do is configure your SSH client to accept only host keys signed by your CA.
However, setting that up is complicated. You need a lot of knowledge to set that up securely. On the other hand, manual verification of host keys is trivial -- anybody can compare a short string of characters.
How you get the public key is up to you, but they are located in /etc/ssh/ on the server, or given to you when creating the server.
ssh-keyscan can scan a host and print the keys as well.
If you also add the servers to .ssh/config you also get tab completion.
It takes a bit more job to do, but it feels much safer afterwards, and it's a good routine.
This vulnerability is only about copying files from the server to the client, and lacking path validation when that happens. This problem can be fixed without fixing MITM.
Sounds to me like everything the user executing the scp command can access can be compromised.
This can only happen if scp is invoked from the home directory (or from root or /home).
sock = connect(server)
sock.do_crypto()
sock.send_filelist(commandline.list)
cd(commandline.targetdir)
while not sock.eof:
write_to_local_disk(sock.receive_file())
Instead of a "filelist", it's got to be a remote shell command argument, since it allows stuff like:
scp server:'$(
comm -23 \
<(find dir1 -type f -printf "%P\n" | sort) \
<(find dir2 -type f -printf "%P\n" | sort) \
| sed "s:^:dir1/:"
)' .
> Due to the scp implementation being derived from 1983 rcp [1], the server chooses which files/directories are sent to the client. However, scp client only perform cursory validation of the object name returned (only directory traversal attacks are prevented). A malicious scp server can overwrite arbitrary files in the scp client target directory. If recursive operation (-r) is performed, the server can manipulate subdirectories as well (for example overwrite .ssh/authorized_keys).
I don't understand why this is considered a vulnerability. The user provides scp server-side shell code to describe the files it wants. How's it supposed to verify object names then? Am I the only one that likes to do things like the following?[1]:
scp server:'$(ls -t | head -1)' .
scp server:'*.pdf' .
scp already describes the files it's writing to stdout. I don't see what more it can do without sacrificing usability.
The possibility of a compromised server writing things that have nothing to do with what I asked just seems like an acceptable consequence for the power scp provides.
[1] - Please, discussions about parsing ls output are besides the point, right now. Heuristics are useful too, at times, for ad-hoc portable practices.
scp server:readme.txt .
If you write:
scp server:'*.txt(oc[1,10])' .
The server description could be a subset of objects it writes to local.
Can someone explain how “employ ANSI codes to hide” works?
echo "Magic!" echo -en "\033[1F\033[2K" echo "Moar Magic!"
This script will print "Magic!" then the next echo sends escape sequences that move the cursor up 1 line with '\033[1F', then '\033[2K' clears the line the cursor is on, which is now the "Magic!" line. Then we output "Moar Magic". 3 commands but only one line of output.
The -e option on echo enables the escape sequences instead of literal chars. -n disables echos automatic newline for the sequence that deletes the first line
scp servers can of course do all the same trickery.
$ echo -e "hello\n\e[1A[2K\rworld"
$ world> Due to the scp implementation being derived from 1983 rcp [1] ...
> [1] https://www.jeffgeerling.com/blog/brief-history-ssh-and-remo...
But its another matter if ansible's scp client is vulnerable to this.
scp_if_ssh = false
Oh whoops wrong context.
$ ping sintonen.fi
PING sintonen.fi (95.216.45.180): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
...
This should be a WONTFIX.
The idea that it could be not only a man-in-the-middle attack but simply a "malicious scp server" is completely ridiculous.
No secure login or transfer method can protect you from the actions of a malicious server, which could be anything.
If I have the root privs to install malicious software on the server, why would I muck around perpetrating a scp exploit on remote users.
Then you're talking about a man-in-the-middle attack, which I acknowledge as a problem.
I don't agree that fixing bugs which date back to the completely insecure rcp program are the key to solving man-in-the-middle attacks in the SSH suite.
If you have to put in defenses at that level, you've already lost; the goal must be to eliminate or minimize the threat of a MITM attack on the authentication system.
This nonsense is analogous to putting a layer of Pig Latin and ROT-13 into the protocol in case the attacker breaks the AES cipher.
1. Use `VisualHostKey=yes` in your ssh config. Learn the randomart image for your server. Don't try to compare two long random strings directly, that's not a job for humans.
2. Use an offline SSH certificate authority to generate SSH certificates, bypassing the need for a host fingerprint check altogether. If you can trust your offline CA, you don't need to trust your server's host key at all. Either they have a valid certificate or they don't. Your SSH client won't even ask you if you trust the server.
It's one thing to access a compromised server, but you should NEVER be exposed to MITM attacks with proper SSH usage, except for your very first connection on a brand-spanking new server when you drop your SSH host certificates (ideally in an automated fashion which can be deployed immediately after spinning up your box).
The CA approach is probably good if you control the servers you use. Right now I connect to about 5-7 ssh servers on a regular basis, and I don't have (full) control over any one of them.
Save this key, and reuse it everywhere before connecting to github.
Now the issue of being MITM'd is once again only an issue with your very first connection, which is done via ssh-keyscan.
As for your other servers, you should only check the host key once for any of them, and then save it. And that's only if they are owned by a third-party, like a shared server. If your provider routinely cycles host keys, get a new provider who actually cares about security.
If these servers are owned by your employer, then your employer needs better security practices and, failing a certificate authority, should provide you with the necessary host fingerprints before you ever connect to a box.
Looks to me like the fingerprint-checking process has to be manual if you can't choose the server certificate.
Try applying that logic to a web browser. Oh, just do whatever the server says and overwrite whatever files it wants?