Sshmuxd: SSH proxy to replace jump hosts
github.com
github.com
We already help with automating jumpbox creation. (docs: https://userify.com/docs/tips/jumpbox/) and we're building even more jumpbox automation now. Don't allow root for any jumpbox accounts, but of course root escalation exploits abound. (I just found another in an AWS agent yesterday.)
Of course, as another commenter mentioned, if your jumpbox is compromised, than the jumpbox could serve as a gateway to your network. It's a tradeoff between exposing all of your servers to inbound SSH or only one.
There's another way, which is pure TCP forwarding on a different port for each server (ie 21321 -> inner server 22), but whether this actually reduces the attack surface is debatable, since the totality of open ports remains the same across the entire network.
My personal feeling is that using a jumpbox and locking it down (preferably to your company's IP ranges, etc) is the best way to go. You can also add MFA to the jumpbox entry point itself. We're going to help with automating all of that in the near future, too.
(disclaimer: CTO @Userify)
One thing is setup of jump hosts (basic version just being a box with sshd enabled + users with authorized_keys, which is beyond simple), but it's the limitation of hosts. I have 1 big network where a set of users are only allowed to jump to 1 or more unique host each, with myself and a few others being able to jump to every single one. I believe that your solution requires things to be on different subnets to isolate them, with separate jump hosts for each subnet, correct? Here, I only need one sshmux for everyone.
Another "fun" feature is that sshmux can also throw unknown users to a separate host, such as ssh-chat, for support or hanging out.
Actually, in case of -oProxyCommand="ssh -W %h:%p jump_host", you're still fully secure in the case that the jump host (sorry for not calling it a jumpbox) is compromised. This is true both for normal SSH servers and for sshmux. They can mess with the original raw ssh connection that requests the forward, but this will just screw the connection up. They can connect it to a wrong host, but that will provide incorrect host keys. The agent isn't forwarded, so they don't get to sign anything with the private key. Messing with the new connection that gets established is subject to SSH's normal MITM resistance. This means that even root on the jump host would require real SSH protocol vulnerabilities to attack the connection.
Good luck with Userify!
> I believe that your solution requires things to be on different subnets to isolate them, with separate jump hosts for each subnet, correct? Here, I only need one sshmux for everyone.
Right, we're automating the .ssh/authorized_keys (and sudoers) right now (https://github.com/userify/shim/blob/master/shim.py), so you can set it up however you like, as long as those boxes can get to either Userify for key updates, or your on-premises Userify server(s). Only one jumpbox is needed across any network, depending on how you prefer to lockdown your network. (The HOWTO refers to EC2, but it can be built anywhere, exactly the same way.)
sshmux looks like an interesting project, and love the simplicity. Wonder if Userify might automate the ProxyCommand setup! Seems really useful.
> Good luck with Userify!
Thanks! We have 350 companies on the platform right now and growing fast, and Userify Enterprise (on-premise) is in beta at several large enterprises as well.
BTW, joushou, we're looking for brilliant SSH and golang hackers.. ping me if you get bored :)
Yeah, ProxyCommand is a bit cumbersome for many setups, which is why the agent forwarding mode has the interactive selection screen...
If you feel like poking at sshmux or think there's anything that would be helpful if added, don't hesitate to add it to the issue tracker. Real-life use-cases get higher priority.
> Thanks! We have 350 companies on the platform right now and growing fast, and Userify Enterprise (on-premise) is in beta at several large enterprises as well.
That's great! I guess getting Userify Enterprise out of beta is a bit thing on the roadmap.
> BTW, joushou, we're looking for brilliant SSH and golang hackers.. ping me if you get bored :)
It sounds very interesting (I like Go and SSH, what can possibly go wrong?!), and I am indeed occasionally bored, but I'm ~8000km away... Commuting from Sweden to Denmark is bad enough! ;)
This is very dangerous.
With the average setup, anyone with root access on the middle box can borrow the ssh key of any user connecting through it.
See http://unixwiz.net/techtips/ssh-agent-forwarding.html#sec for more info.
In the normal case, this is done through a socket, which means that root on the machine can access it, and sign arbitrary things. A compromised machine is therefore dangerous. This is no different than getting root access on the local machine, as you can access your local ssh agent there as well.
However, with sshmux, the agent channel is not forwarded as a socket, but instead internally handled as the ssh channel it really is. This means that compromising the agent at sshmux requires arbitrary memory access, which is a considerably more difficult feat, and means that the machine is 100% compromised regardless.
Of course, without askpass, a properly patched evil sshmuxd could sign a separate session while still signing ths original.
As I have said many times, ssh -W ProxyCommand makes you immune to evil jump hosts, but if root on a machine in your network got compromised, as would be required, then you're going to have a bad day regardless.
I am root, I have arbitrary memory access on the jumphost. I could write a tool, like the heartbleed poc, to find the data I want.
I wrote it because, in the case that you have an evil user that scans all physical memory, you can't do anything on the computer and still be safe. ssh -W bypasses the host, and due to various cryptographic features of that setup, the only thing that could happen would be to have someone break the connection or present you with incorrect host keys, which the client would detect. I'm of course assuming that they don't have a SSH vulnerability, as that's a completely different story regardless of agent forwarding.
You can't use that machine for anything else and still be safe, and the presence of that machine on your network is a bad sign, most likely indicating that other things will be taken over as well in the same way that they got first one, or already is compromised, including some of your endpoints. While agent forwarding to the final host isn't necessary, your network is compromised, meaning that other important thing most likely got stolen, just not your private key. If your trumph card is that they don't have your private key, it's kind of like standing at a house that has been completely destroyed with all valuables in it, holding a key and saying "Well, at least they didn't get the key for the front door!". In this case, having private keys for different purposes would be a good way to isolate things, just in case. I personally separate work keys and private (as in spare time) keys.
I know I'm exaggerating the comparison a bit, but the setup required to take over agent forwarding on a network you control, means that you're screwed regardless. Agent forwarding should be used with care, ssh-add -c being absolutely necessary, but agent forwarding has its uses, and many of the cases where I'd be attacked means that I have more pressing matters to deal with.
This is yet another situation where Kerberos got it right years and years ago, but is almost impossible to actually /use/ even now after n years.
I've wondered from time to time why Kerberos more or less faded into obscurity. I guess we have the answer then.
Kerberos could have bad interface as well, but I don't know how that would work in this scenario. The protocol itself is secure, but just as with ssh agent forwarding, if you let everyone make authentication requests without requiring user interaction to verify it, it will have the same problems.
I'm curious, how did you get that set up?
The project is meant to ensure that you can allow multiple users access through a jump-host style mechanism, while not permitting any other "abuse" of the jump host. You can lock down SSH a lot, but not as much as sshmux does. Security wise, the code is very, very simple and easy to follow, and even if it went rogue, that's no different than the trust you put in your usual SSH server. If this is a concern, do not use the agent forwarding mode, which would render a rogue server a pointless and unfruitful prank.
If you use the ProxyCommand mode, the only thing an "evil" sshmux would be able to do would be to break the connection. It won't be able to fake the endpoint if it is already in your known_hosts, as it does not have the real endpoints private host key. It is, therefore, secure in this mode.
[1] https://github.com/sshuttle/sshuttle
[2] https://groups.google.com/d/msg/sshuttle/jdTzJGjTDMg/IcQkCzb...
As for sshuttle, it's a very convenient device, although the design appears rather complicated. I'll look into it.
sshuttle is:
- a tcp multiplexer (multiple connections onto one stream)
- a router (uses firewall rules to make normal connections go through the multiplexer)
- a few kludgy tunneling operations, to make the combination above seem like a real VPN (discovery of remote subnets, tunneling of DNS requests, ....)
- a default setup that runs the multiplexed stream through an ssh connection, thereby giving all the security guarantees that ssh provides (integrity, confidentiality, mitm resistance)
- packaged in such a way that the remote side needs to have a minimal python>2.6 install, and a user capable of making tcp connections. Nothing more.
In my experience, it works way better than IPSEC tunnels and some commercial VPNs that I've used. With two caveats that I'm ok with with: a) only TCP, and b) all connections seem to come from your remote ssh server. No VPN solution that I'm aware of makes so few demands; do you know of one that doesn't require root at the remote server?
SOCKS5 most common usage is to just use it as a proxy for HTTP traffic or similar, but it can make any connection. The trick lies in the "kludgy tunnelling operations" and the "router" (firewall rules).It's this magic that makes it better than the socks5 proxy solution, as it really does appear like a VPN. I wonder how much effort it would be to get rid of some of the "magic", though.
But I have to agree, that ANYTHING is better than a real VPN in complexity.
No, but it requires a SOCKS5 client, of which there aren't many among software I use.
> The trick lies in the "kludgy tunnelling operations" and the "router" (firewall rules)
The other important part of the magic is the packaging. While ssh+socks support is pretty common these days, it is not supported on e.g. dropbear (which is popular on routers).
sshuttle copies whatever it needs to the other side; as a result, there's never an issue of version compatibility (which socks use sidesteps by virtue of using an old, cemented protocol). I have, in fact, needed to modify sshuttle with new options and features for a project, and I was not bound in any way by the existing protocol - whatever changes I made were always supported by the other end of the connection because the connection starts by copying them to the remote.
> I wonder how much effort it would be to get rid of some of the "magic", though.
sshuttle is very well written; It is easy to tear it apart and rebuild it in other ways. But if you took the magic apart, you'd be left with a non-standard UDP-deficient SOCKS-alike system. Why would you want that?
With that said, I might still end up using sshuttle as it is. It's a nice project. I just crave for tinkering around. :D
I'm not sure I get what you're on about, to be honest. :/
Maybe I should just install and try this for myself...
If I get around to it, one could wrap sftp completely, so that the available servers simply show up as root-level directories.
EDIT: Opened an issue about it.
That wound be amazing! I'll see if I can check if ProxyCommand is doable with FileZilla later as well.
Seems like the same thing. Even written in go, too.
SSHProxy is a two part thing:
1. A server that, when connected to, will ssh to the requested host, but authenticate with a local private key.
2. A client that will, when connected to the server, work like netcat for ProxyCommand purposes.
I'm skipping some info about additional SSH sessions being involved, as they don't provide any additional security.
The result is essentially ProxyCommand="ssh -W %h:%p jump_host", but instead of you having the private key, the jump_host has it. You then use a different private key to authenticate to the jump host.
I get the idea, and I by no means say he shouldn't have implemented it, but this provides no additional security what so ever. It just moves your private key from one machine to another. I hope he had fun implementing it, though! If you don't implement what might be a bad idea, you'll never get around to the good ones. Plus, bad ideas can be fun, even if they remain bad when done.
You are just wrong on the no additional security though. In addition to enabling auditing, it also allows a username/password to be turned into an ssh key, so that you can know that if the rpi wasn't turned on, then nobody could log in to the target machine. It's similar to a TPM in this regard (as blog says), or can be a jumpgate protecting the systems behind it from compromised or malicious users.
I'm sorry you don't get it, but I'm not hurt by someone not understanding something when they criticize it. If you don't see the point of not having direct access to your key, then there's nothing I can do.
1. The private key is not secure by default by putting it on a different host. 2. This still allows an evil user getting access to his original machine to SSH into any host requiring his private key.
1 should be pretty obvious. If my intention is to get a hold of the actual private key file, he just moved the target. As for 2: We're talking about avoiding compromising a private key file stored on the client, so in order for that to be an issue in the first place, we're giving an evil user Y reading abilities on the local client. The normal user is called user X. The evil thing to do is to log into the other server on poor user X's behalf. User X used SSHProxy, and moved his private key to a raspberry pi. So, what does evil user Y do on user X's computer?
He just logs in through the SSHProxy, getting access to the remote service. Having the raspberry pi disconnect at times doesn't really matter, as that just means patience (or starting a job that keeps trying). Once logged in, evil user Y could add another key to the authorized_keys file if he wanted to.
Having a service automatically sign things with private key A stored in remote location B, as long as you provide private key B stored in "easily accessible" location A (Which we still assume due to thinking the private key was insecure) is no more secure than storing private key A in "easily accessible" location A. The only argument becomes the ability to occasionally take the pi offline, which doesn't seem to be what he touts as a reason for the project. The issue with this is essentially a reversed modified version of the ssh-agent socket issue, which is blindly signing things as well.
If you truly want to protect your private key: * Encrypt a USB dongle * Put a password protected private key on it * Only plug said USB dongle in when you need it, store separately from your laptop (on your person) * Change your password occasionally
Likewise, having a TPM be your signer doesn't change anything if you have it sign blindly. PCR's allow you to at least verify system state first, but using the TPM in this case only protects you against people stealing your filesystem without your physical machine. With access to the machine, the TPM will keep on signing things blindly. User interaction beats TPM in all areas. One of the cute features TPM was supposed to bring, that actually was going to matter (if you ask the manufacturers, that is) was Remote Attestation. Unfortunately, it wasn't any good and was never used for anything.
Also, for future reference, while it's completely okay to suggest that people are wrong or misunderstanding, there are nicer ways to say it than "You are just wrong", and "not understanding something when they criticise it". I have no idea about your qualifications, and you have no idea about mine. We're all humans, so be polite about it. No one likes being insulted.
It's useful information to me that you value pleasant phrasing over criticism that might improve your security.
Thanks for making this clear up front.
However I'm not that familiar with go so might be overlooking something. Is there any logging included on who connected at what time from what IP?
It's actually quite common, especially in Malmö (Malmö<->Copenhagen takes about half an hour, Landskrona<->Copenhagen 1:20), for a lot of reasons. The first being that living expenses, including rent and household items, are cheaper in Sweden. Local wages are always adjusted after the local living expenses, so living in Sweden while working in Denmark is economically favorable, especially seeing that you get a tax deduction bonus based on distance from home to work through the shortest feasible route. Cars are also orders of magnitude cheaper in Sweden due to different taxing. Not everything is cheaper or better there, though. Diesel and alcohol is more expensive, and my selection of rye bread is much narrower up there!
For me, it's due to me wanting to start a life with my wife, and a lot private things going on that made Sweden the obvious choice.
In our setup we use jailkit allowing only ssh passthrough [1], we have added LDAP support to it (may release the patch later).
Agent forwarding does not provide private keys, but only individual signing requests. This means that while the user is connected, and evil remote can request arbitrary signing, but only as long as the user is connected, and only as long as the users ssh agent is willing to do so. Using ssh-add -c further means that the user will have to accept each signing request. Also note that ssh -W, which is immune to any of these concerns, is fully supported by sshmux.
Correct me if I'm wrong, but "Jailkit" does not seem to stop an evil legitimate user from poking around other hosts with forwarding and such (which they can as long as ssh -W functions, within the limits of what a firewall allows the jump host to do in general). sshmux allows you to lock users to targets, rather than lock general capabilities of the entire jump host. This is because, while I provide clients access to various hosts, I do not trust them enough to have any unnecessary privileges.
ProxyCommand ssh -q <jumpHost> nc %h 22
Edit: just tried `-W` as an addition to the line but I get
littlefish :: ~ » ssh pgsql3
Bad stdio forwarding specification '<jumpHost>'
ssh_exchange_identification: Connection closed by remote host ProxyCommand ssh jumpHost -W %h:%p
Agent forwarding with sshmux should be safer than the general case, though, as the agent is handled in memory, rather than as a socket. The agent is also not forwarded to the destination.
There is also the user-level access restrictions of sshmux which will be much more difficult to replicate with VPN's. At least with pptpd, I believe it would require putting users on individual subnets with firewall rules restricting access to only their permitted hosts. Long story short, it wouldn't be a feasible solution.