Bastion hosts
pandastrike.com
pandastrike.com
If OpenVPN scares you, you can still do layer-4 filtering to minimize the impact of losing the OpenVPN server (ie, to get it to "no worse than if we hadn't used VPCs in the first place).
"Single-packet authentication" is, I think, silly, and I've never recommended it to anyone. It's hard to think of a case where hand-rolled authentication in front of SSH has saved anyone with a proper SSH configuration (keys-only-no-passwords). You can skip that step.
Is it though? Is having your (and your co-workers) desktops/laptops/and possibly more directly able to access production servers at a packet level a good idea?
I would posit that it is not. A bastion host can be used to lock down more than firewalls, and reduces the attack surface living on your network considerably.
> VPN terminator in the VPC serves the same purpose as a jump box.
It certainly serves the same network separation purposes a jump box does, but is not able to dictate the access environment nearly as much. If only your jump box has access to servers (and production servers properly filtered), every bit of remote access must go through that single ingress/egress point. And that point can have security policy built-in as well for multi-person teams. For example: I can enforce certain key types being utilized on remote hosts, logging of ssh sessions, etc.
I also agree the "single packet authentication" is silly though. Most bastion hosts these days will be behind VPN already. Just another riff on port knocking really.
Most notable of these issues is the nature of either needing SSH agent forwarding enabled or storing a copy of the private keys on the bastion host (please don't do that). There are ofcourse ways of dealing with this but they aren't worth it.
It's fundamentally much better to just harden every system and implement proper federated authentication and authorization with LDAP. It also makes it much nicer to implement things like MFA and device level authentication.
Remember, it's very rarely that an attacker is going to be able to successfully walk in or break down the front door. Your security holes are very unlikely to be in your administrative channels like SSH/RDP but rather crappy applications or services you may be running on your systems.
Those crappy bits of software + insider threat are the 2 main things you generally need to deal with. Ensuring systems are sufficiently isolated from one another helps with both of these and bastion hosts just add another layer of shared stuff between workloads that should stay entirely separate.
I only said bastion hosts are bad and generally decrease overall security because it's almost impossible to craft one that doesn't create more security holes than it closes.
Much better? Yes. More practical? No.
Once you have a network of more than 3 hosts, host security can quickly become difficult to maintain. This means not just keeping up to date on patches but auditing all your configurations and carefully reviewing applications and services.
Every organization in the world has serious host security issues. For example, a company like Google has an excellent security team and presumably world class endpoint security, but there's almost no question that an attacker on some segment of their internal network would be able to pivot and take over hosts (the lateral movement would likely be noticed by their SOC, perhaps within an hour or less, but it would still be after-the-fact).
Proper network segmentation and architecture makes most of those issues much less risky though, as it is much harder to actually get to any of those hosts.
It's fundamentally much better to just harden every system
You must be one of those people that don't make mistakes and only use bug-free software across their entire deployment.
For the rest of us, DMZ's and bastion hosts are the only way to limit our attack surface to something that can be realistically managed.
If you really believe that OpenSSH is too scary to expose to the Internet for fear of memory corruption flaws, don't stop at a half-measure like "port authentication". After all: once you "authenticate" the port, the attacker can simply hijack your TCP connection to trigger the flaw.
Instead, write a plugboard proxy, and have it apply a static-keyed AES-CTR transform to the data.
I think this is silly, but if you're trying to protect yourself from OpenSSH bugs, at least do something that actually protects you from OpenSSH bugs.
I also dispute that a plugboard proxy is "more complicated" than "cryptographic port knocking".
I'm not aware of any available plugboard proxy solution, so it would have the added complication of hacking it together. That's what I meant. And there are diminishing returns there: a non-criptographic port knocking scheme with a few packets could be good enough as a camouflage in many cases. But I still find it to be a time saver in the more general case (not APT). I think having cleaner logs is worth using this simple trick then.
is this because you expect this class of attackers to be able to thwart port knocking more or less trivially ?
I agree. I usually use simple port knocking for that purpose. Surely you can agree that "no ports listening" is a better circumstance than "tcp/22 open to the world, with key-based-auth only"?
A VPN is redundant (and requires special client), whereas port knocking to open a port for ssh (that only supports key-based auth) is a simple matter of a few "nc -w 1" lines in a script before you ssh...
EDIT: Saw your posts below. You presuppose that such OpenSSH-zero-day holding attackers are specifically targeting you, which is not invalid; however a spray-and-pray quickly after public release is much more probable in a lot of cases.
(tls-auth doesn't protect you from vulnerabilities in the OpenVPN data protocol, but that's a lot simpler.)
Then you avoid having a single point of failure. If and when a single SSH key is compromised, there are a limited number of hosts that key had access to - and unless that's the log server, you have secured logs so you can figure out what happened. Different keys can have different capabilities, rather than a one-size-fits-all policy on the bastion. It makes it harder for an attacker to know where to get the boss's keys. And you can expand your system geographically without the performance compromises of a VPN; your servers already make encrypted, secured calls to each other (over SSH, SSL or a very small number of vetted protocols), expose only those ports that are actively used. Your attacker could steal your entire system documentation and it wouldn't make any difference, because the system is truly secured, not relying on obscurity.
That way you can rotate them easily if the private keys become compromised.
I recently built a system using this technique and synced the keys to S3, very effective.
Defense-in-depth is still the only sensible way to handle security. You do what you can on the host side, but bastion hosts, jump hosts, VPNs, etc. provide an additional layer to make lateral movement much more difficult for an attacker.
"You appear to be using a browser that may not be able to display our site correctly. Generally, this is beause it's an older browser that doesn't support a lot of the newer, awesome features in open Web standards.
Rather than provide an untested and possibly subpar experience of our site, may we suggest downloading one of these fine browsers? (We think you'll find your overall experience of the Web will improve if you do.)"
This is followed with links for Firefox and Chrome. I am running Firefox 24.4 from CentOS repos. This is unskippable, and there's no simple way to bypass it on the page. On a browser from March of this year.
Urgh.
curl https://www.pandastrike.com/posts/20141113-bastion-hosts | gunzip | moreThe only two proper excuses for ever using a bastion host are "I don't have the money to do it right" and "I don't have the time to do it right."
OpenVPN with tls-auth switched on is a pretty robust front door. If you combine that with enforced key only auth for ssh, ensure all internal machines run their own firewall and logically split different bits of your internal network off in to their own subnets you've got a fairly secure foundation to build upon. Another easy win is switching internal services over to TLS/SSL only (where it makes sense/is possible to do so).
Rather than provide an untested and possibly subpar experience of our site, may we suggest downloading one of these fine browsers? (We think you'll find your overall experience of the Web will improve if you do.)
As for the port knocking silliness, well, I don't see any reason for it.
http://bsdly.blogspot.com/2012/04/why-not-use-port-knocking....
Say you have a bad actor behind the same NAT'ed IP. Good User A signs their SPA, and the firewall opens up for 30 seconds... during which Malicious User B has access.
What real advantage are you gaining vs. just having restricted firewall rules in the first place? Lowering the window of time an attacker has access? If you're not using password auth that could be brute forced, who cares about the window? Either they have the keys or an exploit needed to get in, or they don't.
In an enterprise setting, this is probably not helpful. At home or for private servers, this is a nice feature.
(Also at home, it helps reduce stupid ISP imposed rules on not hosting ssh... their scanners just assume "no server here" but I can still get to my home network)
Yes, it is a password with a tiny number of "bits". But nobody suggested using only port knocking, so that's not significant.
Yes, if my attacker is sharing the exact same NAT'd IP as me then they have a window to attack ... my fully hardened server, which was no worse than what was presented to them before.
Finally (and this is what makes me happiest) my sshd is not exposed to whatever new exploit might be discovered tomorrow that doesn't require user credentials. It's not exposed to being DOS'd by a buffer overflow vuln. It's not filling my logs with brute forcers all day and night.
I love port knocking.
[1] JohnCompanies (first VPS provider)
[2] rsync.net
philjr, that is just complete BS. I've run SSH on non-standard ports and they still get a lot of attempts. You know, there are these things called port scans that easily find the daemon on alternate ports.
1. When was the last time your sshd was hit with a 0day?
2. Why aren't you running grsec to prevent buffer overflows (amongst other things) ?
3. Wouldn't it be easier to DOS your host with excess traffic (like all modern DDOS attacks), rather than find a remote-code-execution buffer overflow 0day and use it solely to starve resources on a bastion host?
4. Why don't you use a VPN? If you used a udp-based VPN nobody would even know the port was open (connectionless) and you could even require a preshared key, which is the more secure equivalent of port knocking.
Both ssh and a vpn support authenticating with public keys. Both require maintenance on security updates, or to gain new features, or to strengthen ciphers used, or allow a new user access.
You mention certificates. You do not need to use certificates with either, though you can. You do not need to validate certificate expiration with either, though you can. You can also set expiration to some year well past when the technology will even be relevant.
You claim that expiration of a certificate is somehow a flaw, yet it exists as a security feature. If what you want is to be more secure, you would be wise to use this feature, and not discard it as an annoying maintenance task. However, happily, you can easily ignore it with a VPN just as you do with SSH.
Wow. So, in your mind software that is more complex is more secure. OK, I think I understand the origin of a lot of your comments, then.
We completely and fundamentally disagree.
What do you do when your SSH or port knocking service fails?
That's a complete red herring. What do you do when your firewall fails??
What is the state of grsec on FreeBSD ? I'm not sure...
Answer to #3 is yes.
I don't use a full-blown VPN in a lot of cases because it's a lot of complexity that isn't needed. knockd is extremely simple (look at the source) and simple to use.
---
This is the way all discussions about port knocking go:
"I like port knocking"
"But port knocking is not the christ child that will deliver us from all of the evils of the world!"
No, it's not. It simply adds a small amount of incremental security at almost zero cost. All else being equal, my sshd was visible before, and now it's not.
If the port knocking feature exists to "hide" your network service, you're saying the security of your service is based on the idea that nobody can see your traffic.
But if someone could see your traffic, and therefore your connection, and thus the port your service runs on, they could also clearly see the contents of your traffic. So port knocking would be useless, and everyone could see your traffic, and probably be able to do anything they wanted with it.
But since you use port knocking, you clearly believe nobody can see your traffic. So it then follows that you don't need encryption.
So clearly you should be using telnet.
This is just an absurd statement. You have conflated to completely separate issues.
you're saying the security of your service is based on the idea that nobody can see your traffic.
No, it's because no one can find the service, not see the traffic.
1. Anyone who can see your traffic can see where the service is.
2. We assume anyone can see your traffic because you're using an encrypted connection. (The conversation should have stopped here, but i'll continue anyway...)
3. The only thing port knocking does is obscure your service from port scanning.
4. The equivalent method to preventing someone finding your service via port scanning is changing the port number.
5. Changing the port number prevents all non-direct-attacks from finding your service. (Nobody scans every TCP+UDP port of every IP on the internet for one service, because 563.6 trillion network packet send+recvs takes a long time, which doesn't even factor multiple tries or error retransmissions).
6. A direct attack by a committed attacker will eventually find your network service.
7. Port knocking does not add to security. It partially obscures information, but it does not prevent access, and it operates on faulty assumptions about network security; the information is still there, you just have to look for it differently. It is less reliable than even IP whitelisting.
8. In the real world (and not the "gee I think this sounds more secure" theoretical world) every single other network service you have will be exploited before OpenSSH. So trying to hide OpenSSH while not hiding your other services becomes farcical.
Dude, right there you lost me. 99.99999999% of the internet can't see your traffic. The NSA can. Your ISP can. Random threats on the 'net at large cannot see your traffic.
Ways I can see your traffic:
* fucking with bgp
* fucking with routers in general
* fucking with switch port tables
* fucking with proxies
* spoofing
* sniffing wireless networks
* sniffing wired networks
* tapping a cage backbone
* rooting an isp
* rooting a dslam
* rooting pops
* rooting firewalls
* rooting endpoints on management ports
* abusing hosted networks
Don't assume that just because you aren't aware of a particular attack that it doesn't exist or doesn't apply to you. Hackers are only limited by their creativity.
Please don't trust the network.
I love this. I absolute hate VPN. It's waaaay too complex and I fear vulnerabilities in that waaay more than other software. So, I'll stick with port knocking. It's much safer, IMO.
(Your most secure server) is not as secure as (your most secure server with some obscurity added).