Client Certificates at Netflix | Hacker News Reader

Client Certificates at Netflix

scaleft.com

38 pointsrchiniquy10 comments

It's unclear from this and the slides what the UX for the engineers is? The repo contains a bless-client that'll fetch a newly minted cert, but the slides talk about integration with SSO - is there another piece that invokes bless and drops the cert on disk?

https://github.com/Netflix/bless/blob/master/bless_client/bl...

You linked right to it.. what do you mean? $ ./bless_client.py region lambda_function_name bastion_user bastion_user_ip remote_username bastion_source_ip bastion_command <id_rsa.pub to sign> <output id_rsa-cert.pub>

If successful, I imagine it signs the pub key "id_rsa-cert.pub" which you then can use to SSH as you please (until the cert expires).

That isn't the workflow described in the slides. They talk about signing in via SSO to generate the cert, so it looks like this client is just an example of what would be run at the back end of the web interface you sign into (with 2FA etc), to drop the cert. Look at that as a UI as well, it expects the bastion user to know their own IP to put into the command? Nope. That's a command to be run by something else.

But that just brings up more questions. Generally you want as little as possible running on the bastion; it really is just that choke point. So I presume the SSO front end is also not running on the bastion, but in another subnet, then pokes the key back out...somehow?

Author of the blog post here.

I wasn't at the talk, but had the same question.

BLESS, as released, appears to be intended to drop onto an AWS bastion host and recommends an AWS IAM role to authenticate the bastion to the Lambda function that acts as the CA. Like you mention, there are a lot of limitations to that, but its still a neat demonstration of using ephemeral client certificates without having to spin up a bunch of infrastructure.

At ScaleFT we're building a solution on the same principles that offers some of what you're talking about out of the box:

  1. Authenticating users against a variety of SSO systems
  2. Treating bastions as nothing more than an untrusted TCP proxy

For me the critical takeaway though, is that whatever automation you're layering on top, client certificates make great ephemeral access tokens and they're increasingly catching on. Its just a matter of building the right automation, integrations, and tooling on the client to make the experience good.

https://speakerdeck.com/rlewis/how-netflix-gives-all-its-eng...

That's a great slidedeck, looks great and I can actually learn something from it just reading the slides.

I'm not sure how much BLESS really improves the overall situation. It just shifts the problem from securing SSH secrets to securing AWS/IAM secrets, which seems pretty much the same to me.

The AWS/IAM secrets wouldn't be passed around, except on initial creation.. Not sure what you mean?

The problem they are fighting is host-level SSH authentication. Developers tend to have SSH keys stored all over the place, and it's near impossible to know if they protected their private keys with a password, or if they ever accidentally published them (how many private keys can you find on github these days?

Using client certificates instead, the developers have to ask for a certificate (presumably with some BLESS client), and after authenticating (with their credentials + 2FA) get issued a SHORT-LIVED certificate (5 mins) signed by your own internal trusted CA, to use for SSHing into bastion host or whatever.

So now, there is no key material handed to developer which will be trusted for more than a few minutes. Even if their password/secret were leaked, you still need 2FA and network access to get a client certificate. If you leak the client certificate, unless someone snatches it up within it's validity period, still no worries.

Author of the blog post here. You totally nailed it about the problems with SSH keys.

I think what zokier is referring to is that the BLESS client uses an AWS IAM role to authenticate users, so if you wanted to invoke it directly from an end-user machine they would need their AWS credentials on disk there. Instead BLESS seems to be intended to be invoked from a jump host which has a machine role giving it access to the Lambda function - which gives you a nice choke point for auditing, but obviously has some other limitations.

I get the impression from the slides that Netflix might have some additional internal tooling for authenticating users against SSO. I'm just excited to see client certificates taking off as a form of ephemeral access token. I see too many organizations that require MFA to access the corporate wiki but let users authenticate to production with a private key from 3 jobs and countless laptops ago.

For a layer of security, when using the client you could do IAM role assumption in your local machine. You have your static creds and an MFA device, then use those to export temporary credentials to actually run the bless client. http://docs.aws.amazon.com/STS/latest/APIReference/API_Assum...

Some tools to make the process easier: https://github.com/remind101/assume-role https://github.com/trek10inc/awsume

SSH secrets live on a developer's machine, whereas IAM backed by 2FA requires a much more complicated attack vector than simply stealing a key file.

Reply on news.ycombinator.com