Hacker News Reader

Comment by idlewords | Hacker News Reader

I have two dozen co-located servers to log into. So far I've resisted the idea of setting up a bastion host, because that seems like a way to lock myself out of SSH access if that machine dies. I'm also not sure of the rationale for having a bastion host, other than "big companies do it". How do other non-cloud people handle this access vs. security tradeoff?

I'm also curious what people's preferred fallback method is for preserving access to machines if you lose access to your yubikey(s), assuming you keep your private SSH key stored on one.

View on HN

> I'm also not sure of the rationale for having a bastion host

It's a central place where you can do your logging, which many enterprises must do for compliance reasons.

Really? Do any compliance standards specifically require this? Or is the specific control defined in the organization itself that requires this?

I've found many compliance standards to be pretty open-ended and function-driven, rather than prescribing specific standards. Client contracts.... they may be a different beast, and often require specific promises by vendors.

PCI-DSS comes to mind and it’s segmentation guideline, often met by deploying bastions.

I'm pretty sure that at least some of the German financial regulations require that you can audit administrative actions on systems that are relevant for the financial transactions.

Now, you could try to implement that on the systems themselves, but how do you establish a mechanism that logs actions of an administrative user, and at the same time cannot be disabled by the same administrative user?

So instead you use a shell control box as a bastion, and regular administration of the target hosts don't have root access to the shell control box, so they cannot circumvent logging.

I haven't yet any standards that explicitly demand bastions, but it seems to be one of the standard implementations that have proven to pass audits, and so it's a pretty low-risk implementation of the logging requirements.

Easy: on each host you log into, you enter into authorized_keys your Yubikey RSA pubkey and a standard software SSH pubkey. The private key for the software keypair, you stick on a thumb drive and keep in a drawer or a safe or whatever.

Later

Sorry, I see below you're thinking about using Yubikeys in addition to keypairs. This obviously doesn't answer that question.

No, your original answer was what I needed! I'm fine using just keypairs where the private half has never left a yubikey.

If the yubikey dies, either another person can provision your replacement, or a configuration management system can do so.

We’re doing this successfully where I’m working.

I'm not sure I understand what you mean. I run a one-person business and my servers are 110 miles away in Sacramento; what do I do if my yubikey breaks?

I think the SOP is to have a second Yubikey registered and stored in a safe place and/or a set of one-time codes that you can use instead.

I actually had no idea SSH had support for one-time passwords, but I see docs for OTPW here: https://www.digitalocean.com/community/tutorials/install-and...

This looks like exactly the answer I needed; thank you!

It sounds like you're trying to do something like Yubikey OTP or U2F with SSH. Both are doable, but I don't think that's the normal way people use hardware tokens with SSH; what I've seen much more is that people use Y4 keys (which are basically pocket HSMs), generate RSA keypairs with them, and configure SSH to accept the public key on the token.

This doesn't require any support on the serverside; as far as the server's concerned, the Y4 is just another RSA key.

It does require the more expensive Yubikey, but there are some security advantages to using it.

No, my thinking is along the lines of what you're describing (generate the key on the Y5 Yubikey so that it never leaves it). The discussion of one-time stuff is just as backup for scenarios where I lose the key, which I invariably will, or need to give other people "break glass in case of emergency" access to the servers.

> something like Yubikey OTP or U2F with SSH. Both are doable

I would be interested in how you think U2F with SSH is doable. Maybe with a custom SSH server and client?

Have you actually see this done in anger?

ssh server trusts internal CA, an application that requires U2F login can issue 5 minute certificates signed by the CA, the ssh session does not close when the login cert expires. See https://github.com/Netflix/bless and https://github.com/gravitational/teleport

You should already have a full backup of your (private) RSA key anyway on an encrypted USB drive in a desk drawer somewhere, so you use that until you can provision a new Yubikey.

It's key to setup two or three bastion hosts with similar diversity to your prod fleet (are you in multiple locations? put a bastion in at least two of those; if not, do you have multiple racks? put a bastion in at least two of those, etc) -- you will have network incidents where some hosts are unreachable, you don't want that to hit all your bastions or you'll be very sad.

Bastions are nice, because you can harden them, or put 2fa on them, and not worry as much about the ssh config on the other hosts -- just make sure they don't accept ssh connections from the outside world.

For anyone interested in trying a bastion host approach, here's a step-by-step guide to spin up a bastion host and enable ssh logging: https://www.strongdm.com/bastion-hosts-with-audit-logging-pa...

I’ve only ever seen this handled with VPN, and the answer there is that you’d need to launch another vpn server in the right vpc/subnet if one went down (or have failover setup).

It looks like there’s some benefits with bastion hosts around locking down commands & access in a fiber-grained way from vpn, but man what a pain in the ass. I can see a certain perspective that would really care to do things this way but VPN from my cold dead hands.

backup key on offline usb. Safely stored in a safe, never to be used except in emergency.

Run 2 bastion hosts?

VPN

Reply on news.ycombinator.com