Hacker News Reader

Comment by jvehent | Hacker News Reader

There are many solutions out there that attempt to solve the secrets distribution problem. Vault is the newest one. None of them solve the fundamental problem of the secret zero: provisioning the first secret on an untrusted system the automated way is hard, and without that secret zero, there is no way to authenticate the new system.

With Vault, you still need to figure out how to push secret zero (here a client authentication token). If you can solve that, why not push the secret you need to push directly, and bypass Vault entirely?

View on HN

We worked with a handful of companies to help us design the "App ID" authentication backend that does exactly that: https://vaultproject.io/docs/auth/app-id.html

It allows you to have a non-sensitive single factor within things like configuration management, and have the second factor come from a machine-local location (instance ID, MAC address, TSM module, etc.). The idea is that another team out of band sets the 2nd factor that developers and config management never get to see. The result is you have full automation without secret zero issues.

As an additional security parameter, you can bind the two factors together to a single CIDR block, so you can have an additional factor restricting access from that machine's IP.

The URL itself above goes into a lot more detail, but our beta users are successfully automated in an elastic environment this way.

That's definitely a trade-off compared to strong authentication, one that I wouldn't want to make in critical environments. But indeed a very nice feature for 99% of low/medium risk applications.

hard problem, but you clearly address it.

There is a typo though: "in client" should probably read "if a client" or "if the client"

So if I read that correctly, the UserID (I find that naming confusing since users here aren't necessarily humans) creation has a human in the loop to add the UserID to Vault and therefore make the authorization decision?

Using an instance id or MAC address as a secret 2nd factor seems surprising to me. This is what banks are asking for? It's nice that there are two factors to the authentication, but neither one seems to be securely random.

No, it is usually one of those values salted with a value that is only available offline.

We've had the same issues at CoinJar, and we used SSH agent forwarding to solve it. This is how it can work:

Every new instance of application server is provisioned automatically by a trusted server (which holds the key to a credentials server). During orchestration/deployment, the application server has a temporary permission to fetch the secrets it needed, through SSH agent forwarding. Once the deployment is done, the session will end and the application server can never read new secrets until the next deployment.

This way we don't really need a solution like Vault. SSH is mature enough to provide authentication, and encryption is easy once you've figured out how to distribute keys automatically.

How does SSH agent forwarding work if there's no interactive user driving the process? Otherwise the concurrency is going to be limited to however many ssh sessions the client can spawn; and the orchestration/deployment cannot be a handsfree process.

I'm not the person you're asking, but in our system, our secret access is tied to an SSH identity, not an SSH agent. This means that you have flexibility to require a personal user's ssh agent to be present for access to some things, or an identity file (optionally encrypted) stored on the server for less-sensitive things. In the case of things that need to happen unattended, we use an in-the-clear identity file, which might seem a little counterintuitive since security-wise, it's basically equivalent to just having the secret sitting there, but the indirection adds ease of rotating secrets along with revocability if we were to suspect we lost control of such a machine. But for most of our secrets, we tie them to individual user's identities which must be forwarded via an agent. We consider this an acceptable constraint because deployment and app restarts are always driven by a human in our operations.

It's not a perfect system or a panacea, but it's far better than having a bunch of passwords directly stored in config files, in my opinion. We also have a wrapper tool that uses your SSH agent access to fetch secrets for the mysql suite of tools as needed, so nobody is ever tempted to create a personal .my.cnf on prod servers. Anybody who actually has access to a given mysql password is still on the honor system not to fetch it and write it down somewhere, but removing the need to ever even see them eliminates the temptation to put it in a .my.cnf or in your clipboard, which eliminates a whole class of mistakes you can make in handling them.

This is similar to what we do at my company. I wrote about it a bit here: https://news.ycombinator.com/item?id=8826300

Would be interesting to hear more details of yours for comparison.

Why not do it at the network level? In some infrastructures this is really easy. For example in AWS you can set up all your servers in a VPC. Then put Vault on a machine on a specific subnetwork, and firewall it off such that only the machines from your VPC may access it. You have now created a walled garden where anything inside your VPC has access to the secrets without holding an authentication token, while anything outside does not.

Personally, I think this is your only line of defense anyways. Someone with access to your private network will likely be able to do some harm, and if they get even a non-root user on your server, it's game over, since you are just one local exploit away from them gaining root. Remember, once they gain root on one box, it's just a matter of time before they replace /usr/bin/sudo and /usr/bin/login with a script that emails them your password and/or steals your ssh-agent socket.

Because you want authorization and authentication by application. So a web server / proxy needs the public TLS cert but maybe an application server doesn't. Meanwhile the app server needs DB credentials but the web server doesn't.

In your scenario, if something happens to a front facing web server, nothing prevents the attacker to just enumerate all secrets and retrieve them, whether the server should have had access or not.

Defense in depth is a thing.

I believe what you are describing can be built using the primitives I described. Vault, or another secrets manager, would only release secrets to severs on specific subnets. The front-end would be on your front-end subnet, the application server would be on the backend subnet, and the application server would not be authorized to access the private TLS key because it's on the wrong subnet for that particular secret.

To some extent, that's true. The granularity is a bit lacking however. A bigger worry would be that the ability to spin up an host in a specific subnet is the effective authentication. It's not something I'd be comfortable with myself but I guess some people would be.

If I get on your subnet with a host on which I have root, I can do a few things. First, I can now see any unencrypted traffic. Use MySQL? I can read that or even MIIT it. Use unencrypted LDAP? I can do the same. Basically you either assume your VPC is secure or it's not.

This goes doubly so for servers on your network. Either, you assume that only authorized users have access to it or you don't. You have to draw the line in the sand somewhere and say "past this point it's secure".

Now, I am not opposed to layers of security, but there are multiple options for how to define those, and assuming your network is secure isn't a far stretch on some platforms.

Hmm, how many networks these days are unswitched? How are you going to see traffic that the switch doesn't send to you?

Active attacks, like ARP spoofing? Sure, there are environments where that won't work or will quickly be detected, but in many it will.

There's a lot of variability in cloud architectures, and for that reason there's really no one-size-fits-all solution for machine authentication. Sometimes using factors like the network subnet for machine authentication makes sense and adds something; sometimes it doesn't. It's definitely a factor to consider though.

Another example : in AWS, IP spoofing is not possible (according to https://ip-saas-infopark23444378-cms.s3-eu-west-1.amazonaws....). So IP address is a reliable identifier. However, IP addresses of AWS instances are sometimes stable over time, and sometimes not. It depends on how the instance is configured and used. So again, IP address is a potential authentication factor, but there's no "one true solution".

And even once a machine is authenticated, it needs to be associated with some kind of role (or a role-like thing). This part is actually even trickier.

Also, when you are talking about container-based cluster deployment, the network (IP addresses, VPC, subnets etc) is useless for machine authentication, because any container in the cluster can run on any host machine at any time. As an example, Heroku is basically a giant compute cluster in a single security group (for each region).

They used to publish the security group ids, so that you could whitelist inbound traffic from Heroku. Now they don't do that any more, which makes sense because really, what is the point of whitelisting all of Heroku? :-)

Reply on news.ycombinator.com