mmalone

If you have a group of services running inside a perimeter with secure entrypoint(s), issuing client certificates allows you to coarsely segment service-to-service interactions within that perimeter to contain blast radius from a malicious insider or service compromise.

You don’t need revocation because the credential is useless outside of the perimeter. If you find a malicious insider or a compromised service, you fix the service or lock the insider out of the perimeter. You’re glad that you had client certificates in place as it reduced the scope of the compromise. It would be nice if you could actively revoke any compromised credentials, but it’s not critical because you’ve locked out the offending user and/or fixed the compromised service.

In my response from three hours ago about threat model I said “service authentication helps control the blast radius if someone gets inside your perimeter”. I’ve been saying that I think we mostly agree for a while.

If you have a perimeter with well-defined entry points (e.g., a bastion for HTTPS and SSH) then you can lock people out at those entry points. Mutual TLS is defense-in-depth, and how you implement least authority for services inside the perimeter.

I know that CRL is literally active revocation. If you want active certificate revocation you can literally do active certificate revocation. Whether you need it depends on your threat model. And it’s not actually that hard in many scenarios. It is harder than not doing active revocation, and if you find yourself needing it to satisfy your threat model you probably should consider other alternatives for client authentication.

I’m not telling people they should use lots of mTLS but not worry about revocation. Any content I have written that discusses short-lived certificates is very careful to point out the fact that a compromised certificate will be considered valid until it is revoked. Most people who are using our open source toolchain are actually using it for internal server auth TLS, for narrow use cases like issuing certificates for database authentication or VPNs, or for mTLS to satisfy the sort of threat model I’ve described here.

We haven’t implemented anything around active revocation yet. But it is planned, because it is necessary in many scenarios.

If you’re worried about keeping a simple web service running 24 hours a day then you shouldn’t use our stuff. If you’re far enough along to need our stuff, you’ve figured out how to keep a web service running... you have to keep your own web services running 24 hours a day otherwise your microservice system is already going to have availability issues.

The certificate is still containing blast radius when the attacker is in your network and limiting the insider threat from people who have access to some subset of production. Without a cert, they’d be able to access everything. With a cert, they can only access stuff that the identity bound to the certificate can access. It’s doing precisely what I want it to do.

If you find someone malicious in your network, you lock them out of your network. You don’t need to revoke a certificate to do this.

OR, you can do CRL. This isn’t jazz hands, it’s a choice. As you push mTLS harder, and need things like active revocation to satisfy your threat model, it’s worth reconsidering whether mTLS is the right choice. Sometimes it is, sometimes it’s not. Sometimes you don’t have a feasible alternative.

You’re putting words in my mouth again. I didn’t say those were the only two options. I asked what your silver-bullet alternative to mTLS is that has no issues of its own. The position seems to be that mTLS has problems, so don’t use it. But everything has problems.

Re: revocation... the point is that you can revoke attacker access without revoking a certificate.

Or, if you want, you can revoke certificates. I’m not totally opposed to that. CRL is finicky, but architecturally it works well with short-lived certs and it’s not very different than secret rotation. If you can push new secrets, you can push a CRL. In fact, if you can push new secrets, you can also push new roots.

There are many people who use mTLS. Along with consistent service logging, it’s one of the main reasons people use service meshes. It’s also very common in IoT.

It's not intended to be sleight of hand. Confidentiality is the gateway drug. The argument is that, once you're using TLS for that, the incremental cost of adding client certificates and doing mTLS to reduce blast radius is lower. It's not zero. It is lower.

I'm also not arguing that some active mechanism to immediately remove attacker access is unimportant. Of course it is. I'm arguing that active certificate revocation is not the only way to achieve that. I gave two examples: remove access at the perimeter or push new policy to revoke access for the compromised entity. Most microservice systems already have tools here: they're typically adding mTLS on top of a perimeter model with decent privileged access management.

This is a bit of an asymmetric argument, in any case. I have all of my cards on the table. The opposition isn't really proposing any concrete alternative, so it's hard to make a meaningful comparison. If you want active revocation, implement active revocation. It's not easy, but OCSP and CRL exist and they do work. I even suggested a way to do it well: push a CRL to a cloud bucket, which is scalable and highly available. You can call that "cargo culting all of the idiosyncracies of the WebPKI", but I still haven't heard a concrete alternative. So where does that leave us?

The only concrete thing that's been proposed... that I had to propose myself, since no one else would even answer this very simple question, is macaroons. Which I've said repeatedly I think are a good idea. However, it's abundantly clear to me why people, in practice, are choosing client certs over macaroons. Macaroons can't layer into an existing system the way that mTLS can, and if all you want is "blast containment" you can get that with a fairly simple client certificate setup and a bit of coarse authorization.

Regarding CA availability: if you can't remediate an availability issue with a shared-nothing web service in 8 hours (the downtime you'd need for a cert to expire in a default step-ca setup) then you've got bigger problems than certificate management. The 24 hour default is long enough for people to remediate an outage and short enough to get people operationally comfortable with the idea of short-lived certificates. If I were to use shared secrets instead of asymmetric keys, and I created a system that rotated all of my service-to-service shared secrets automatically every 24 hours would you say that's a bad idea?

Someone just sent me a tweet from @halvarflake where he asked for the threat model under which service-to-service TLS is the best solution. I wanted to cross-link these threads since I think they're related: https://twitter.com/halvarflake/status/1337308903512678401

First, agreed. Certificates can't and shouldn't attest to the user's identity. You need another mechanism for that.

If you're securing requests, then securing the channel might be redundant. However, I do want authenticated service identity (not just user identity). The shape of policy I want to enforce is: "<service x> should be allowed to make <rpc> to <service y> in the context of responding to <request> from <user z>". I need the authenticated identity of both the service and the user to do that. Either that, or the authorization could take place higher up in the stack and I need a capability token passed through. Even with a capability token, it's nice to know the authenticated identity of the calling service for other purposes, like audit. Not to beat a dead horse, but I realize that macaroons satisfy all of these requirements :).

I also want confidentiality. Macaroons don't do this. This is an important point that I want to make more strongly this time, because I feel like it's getting lost in the noise. Authentication isn't the only thing people are using TLS for.

I appreciate your perespective. I think the best way to respond is to actually answer @halvarflake's question: what's the threat model that justifies service-to-service mutual TLS?

One answer is: that's the wrong question. Many people use mTLS for flexibility, not security.

Most microservice systems are born as a bunch of web services thrown in a VPC (or, more recently, a kubernetes). They can all talk to each other. Some of these systems are successful, and eventually outgrow their perimeter. Sub-components need to be able to communicate across untrusted channels (e.g., the web). TLS can give you a "cryptographic VPN" to do this.

There are a lot of VPN-y solutions with a lot of pros and cons. TLS isn't always the right way. The big pro for TLS, however, is that it works everywhere. Even if you don't control the host stack (e.g., functions as a service) or if you're on a constrained device, etc. TLS is also baked into most infrastructure: proxies, queues, databases, etc. You can easily terminate TLS easily at your perimeter if that's what you want. Alternatively, you can connect directly from a service in one perimeter to another service or to a piece of infrastructure like a database in another, without a proxy. Again, TLS is ubiquitous and it's flexible.

At this point I think we're at your "you must be this high to get on the ride" model. Why go further? This is where it makes sense to start talking about threat model.

First... I already mentioned this, but, once again: confidentiality. I think this is less controversial, and it doesn't require client certs. So I'll not elaborate further except to say that people need it.

The interesting topic is: service authentication. Why use client certificates for service authentication? I contend that service-to-service mTLS plus a bit of simple, coarse authorization actually does address a number of common security threats. Specifically:

* Malicious insiders: if you're using microservices and practicing DevOps and agile and have engineers on-call you probably have a lot of people and a lot of code that can get inside your perimeter. Coarse segmentation of service-to-service interactions can help here. An engineer who has the ability to `ssh` or `kubectl exec` or tunnel into your perimeter won't immediately have access to every sensitive service you run.

* Lateral movement: very similar to the above, but this time a service is popped or an engineer's laptop is compromised. Service authentication helps control the blast radius if someone gets inside your perimeter.

* Credential leaks: secrets are hard to manage. They get committed to repos, accidentally logged, dumped in error messages and served to end users, pasted in to slack channels, walk out the door with dismissed employees, etc. Using certificates with a relatively short lifetime gives you assurance that compromised credentials will be eventually rotate once an attack vector is closed. I contend that it's actually a lot easier to set up client certificates that automatically rotate than it is to rotate secrets for all of your proxies, databases, services, etc. An asymmetric crypto key is also inherently less likely to leak to logging infrastructure or end users than a password or bearer token since it's not actually in any requests.

Request authentication could mitigate these threats, too. And, again, carrying an authenticated end-user identity through the stack to downstream services is valuable. But, in a microservice system, I'm less worried about code lying about the end-user it's acting on behalf of than I am about malicious humans. If I already have TLS for "my VPN", authenticating clients and coarsely authorizing service-to-service interactions starts looking pretty attractive.

Regarding compromised keys, certificate lifetimes, and revocation... that's all important, but less critical to the threat model than it may appear. If a key is compromised I'm still much better off than I was without my coarse segmentation: the only stuff you can access is the stuff that the service is authorized to access. If an engineer exfiltrates a key to "test something", that key will eventually get rotated, even if I never know it happened. Same with an attacker exfiling a key: they have to maintain a presence on compromised systems to exfil new keys as they rotate. (FYI: by default, the smallstep toolchain issues certs for 24 hours. Personally, I'd push that down to a few minutes.)

It's also worth remembering that we still have our perimeter (or perimeters). If we find an attacker, we can lock them out at the perimeter instead of revoking a certificate. Even if you can TLS through the perimeter, we only need to enforce revocation at stuff directly exposed to the internet (e.g., in a proxy, not necessarily in every service). Active revocation is nice. But it's not necessarily critical. As another example, since you mentioned SSH: smallstep's SSH product uses certificates without active revocation. We revoke access in policy, by pushing ACLs to hosts and locking users out via NSS and PAM. The certificate may still prove I'm "mike", but mike doesn't have access anymore.

Why not do all of this host-to-host instead of service-to-service? Well, depending on your requirements, that may make sense. Certainly in kubernetes-land the line between "a host" and "a service" are blurring. TLS still has a few advantages:

* It works in places where you can't control the host (e.g., functions): it's a consistent mechanism that works everywhere

* It makes applications more self-contained and, therefore, more portable

* It's logically closer to what you want: you're trying to authenticate services not hosts

* There's more deployment flexibility: you can terminate in a sidecar or in the application

* It's easier to pass authenticated identity information into the application

* It works for authenticating to infrastructure: proxies, databases, queues, etc (again, it's a consistent mechanism that works everywhere)

Fin.

(Are we using macaroons yet?)

You’d use that for service-to-service stuff, to mutually authenticate services (not users), as well?

I’ve always thought of AWS IAM as: great for my-service-to-cloud-service, not so great for my-service-to-my-service.

I haven’t looked at AWS IAM in a while. I’m gonna take another look now :)

Woof.

Literally nobody likes x509. Is anyone seriously working on an alternative? I think there’s room for a more modern, simpler cert format.

Sort of relatedly, I actually really like JWK for representing keys.

Do you think a professionally built solution that gets the details right is possible?

Appreciate the thoughtful response!

Reading it over, I think we mostly agree on the facts. It's easy to do mTLS and x509 wrong. The question, then, is what's easier / more secure: doing mTLS/x509 right or doing something else? I think that's somewhat subjective: it depends on your requirements, your environment, and your skillset.

One point that I'd like to reiterate is this: if you want a consistent cryptographic solution that works everywhere, TLS is pretty much your only choice. You could use something else for client authentication, but you probably still need TLS.

As a strawman, here's a sketch of how I'd recommend doing TLS in a microservice system. I consider this "right" for most garden-variety microservices-in-cloud scenarios and don't think it's particularly hard to do. Most of this is already implemented in https://github.com/smallstep/certificates:

  * Deploy the root cert via automation (so it's quickly rotatable) and/or keep it in a managed HSM/KMS. You might harden root rotation a bit by signing your new root with your old root. But, generally, trust config management or container orchestration to push root(s) (you already trust it to push code and secrets). Root rotation (and, thus, bulk revocation) is now as fast as secret rotation (secrets are generally pushed the same way).

  * Issue short-lived certificates per logical entity. If it gets a box and a name in your architecture diagram, it should get an identity and each instance should get a certificate. Use domain names and email addresses that you control for names. Keep certs simple: one SAN. Certificates bind a name to a public key. That's it.

  * Automate certificate issuance. ACME can work for this, but there are other options (single-use tokens issued by config management, cloud-managed instance identity documents or service accounts, an existing device certificate issued by a manufacturer, etc.)

  * Automate certificate renewal. A simple mTLS HTTPS request works for this. This is easy to implement and easy to scale out with multiple intermediates. "Revoking" a certificate just marks it as "not renewable". To reduce risk of outage, in this architecture, it's safe to renew an expired certificate as long as it's not revoked (ACME-STAR basically does this, but it's push instead of pull).

  * If you really need active revocation, fine. One good solution is to push CRL to a cloud storage bucket. Short-lived certs will keep your CRLs small. If you need to do a mass rotation, rotate roots (push new root, wait for rotation, pull old root).

  * Use secure NTP for time.

  * Index issued certificates. CT (trillian) is cool if you want to be fancy. Your existing database or SIEM also works. zcertificate can parse x509 and output a JSON representation of a certificate that you can map to something like an Elastic Search schema: https://github.com/zmap/zcertificate

I want to respond specifically to your first and final points.

On your first point: I understand that in theory an attacker could slip a request across a secure channel, and binding authentication to a request could in theory prevent that. I don't understand how that's likely to happen in the context I'm thinking of here. Which may be different than the context you're thinking of. So let me clarify.

Suppose I have `<end-user> -> <service-a> -> <service-b> -> <database>`. Let's focus on `<service-a> -> <service-b>`. I don't see how using end-to-end mTLS, terminating in `<service-a>` and `<service-b>` application code, would be any more vulnerable to this variety of attack than an HTTP Basic header like `Authorization: Basic base64(service-a:password)`. Surely, the logic in `<service-a>` is simply "insert HTTP Basic header into requests on their way out to `<service-b>`". It doesn't matter if we're authenticating the request or the channel. If you're able to smuggle something malicious into that request, it's gonna get sent over to `<service-b>` with proper authentication attached.

Are we talking past one another? Are you trying to make `<end-user>`'s authenticated identity carry through `<service-a>` to `<service-b>`? If that's the case, then yes: I see what you're saying and you shouldn't use mTLS for that. I'm not sure if there's a term-of-art here, but I call this "end user identity propagation". You need something like a top-of-stack ticket service (a bearer token) for that. Or, better yet, macaroons. I consider those two separate things, though. mTLS is for authenticating your immediate peer. For end-user identity propagation mTLS is a poor choice.

On your final point: you could, in theory, express claims in x509. I'm sure you're aware, but it's been tried before (e.g., SPKI/SDSI). However, I agree that, unless you really know what you're doing, x509 is too complicated for that. Don't do it. You'll likely screw it up. If you're parsing x509 and ASN.1, you're doing it wrong. If you're processing strings that you've extracted from a certificate, and you're not in the habit of writing your own formal languages, you're definitely doing it wrong. Just put a flat name in a SAN. The only thing you should ever need to do with that string is an exact string comparison. If you need to know roles or groups or some other metadata look them up in a database.

(Or use macaroons)

Yea that's reasonable, but there are pros and cons and it's also definitely an opinion :).

I don't see the problem with piping help text through a pager (that's literally all `man` is doing). Alternatively, a tool can use a pager automatically (as `git help ...` does). So `man` isn't required to have a pager. Whether you should use a pager by default is a different argument.

That said, I find pagers annoying because it's hard to switch back and forth between typing a command and the docs. That's way easier to do with scrollback. Although someone in this comment thread also mentioned the `PAGER` env var, and that you can set `PAGER=cat`, which is a good pro-tip. So, TIL.

Still, if you're concerned with command-line ergonomics for a broad audience I think there are more people that either 1) know how to `| less` or 2) prefer or don't care about a pager than people who are gonna automatically just know they can do `PAGER=cat man <some-subcommand-of-my-cli-tool>`.

Plus, there's still the cross-platform argument.

Oh, and OP doesn't mention this, but man pages also complicate install, uninstall, versioning, and having multiple versions of a tool installed simultaneously. Built-in help is obviously just works and is distributed with the standalone binary. To use man pages you have to place additional artifacts on the system and make sure they're managed correctly (e.g., versioned). It's additional complexity with basically zero benefit.

So yea. Seems like, as a rule, their guidance is reasonable. I don't see a lot of benefit to using man pages other than "because that's how it's always been done".

They're not saying you shouldn't provide command-line help. Just that you should deliver them through a `help` subcommand and/or through a `--help` flag (like `git` does) because people don't find man pages and because man pages don't work on every platform.

Yep. Not perfect... type systems have come a long way since PowerShell did it. But it's a good start.

Look at stuff like PowerShell secure strings for an example of the sort of neat stuff you can do.

Yea, like, imagine if posix pipes and command-line flags could be typed.

Recent macOS versions don't have `ssh-askpass`, and it's weirdly hard to add one. Since agent confirmation depends on askpass, I don't think there's an easy way to get this work on macOS.

Aside from the missing context you mentioned, the other bigger problem with this approach is that agent confirmation is all-or-nothing: it turns on confirmation for local SSH connections in addition to forwarded connections. If you're using SSH a lot, having to confirm every connection is very annoying.

YES. It would be so easy for OpenSSH to fix agent forwarding. Just need to limit authority and/or ask for consent for a particular action.

I wish there was a better way manage telemetry for open source. We have this problem at smallstep, too. We don’t have any telemetry — nothing is instrumented / nothing phones home — but that means it’s really hard to know which features are popular. Which makes it hard to prioritize. We only find out when something breaks and we can look at the volume of issues, heh.

We ask people to let us know if / how they’re using our stuff, but very few people do. It’s a tough problem.

In general I'd say SSHing from a remote box to another remote box is a non-issue since you can always use some sort of tunneling/bastioning to make that work.

It's when you want to use some other tool that tunnels over SSH -- scp, git, sftp, rsync, etc. -- that you run into trouble.

Consider a remote development instance. So I'm an engineer, and instead of developing on my local I do development on an EC2 instance in AWS (there are a variety of reasons this may be preferential or even required). If I'm pushing & pulling code from GitHub to/from this development instance I'm gonna need to either have a private key on that development instance or have agent forwarding back to my local.

Now suppose I want to copy a large database snapshot between two instances on EC2. There are obvious benefits to being able to transfer directly between these two instances, on the same private network, versus bringing the data to your local (where you might not even have sufficient disk space) and pushing it back up.

Not sure what other use cases exist, but these two alone I think justify the existence of agent forwarding.

SSH agent forwarding does have subtle security issues. But so does putting a private key on a remote server. If you're the only one who has access to the remote server, neither option is problematic. If someone else has root, and you don't trust that person, then both are problematic (and which is worse is debatable).

If you like ssh-import-id to pull keys from GitHub, you’ll love AuthorizedKeysCommand to pull keys from GitHub.

Depending on use case, though, this can be a bit sketch. At smallstep we like SSH certificates, which make life similarly easy on everyone with a bunch of other benefits. You can find a couple relevant posts on our blog if you’re interested.

Incidentally, GitHub now supports SSH certificates (for enterprise edition, at least).

Yea we like certificates at smallstep. We’ve got a couple[1] other[2] posts[3] that cover them pretty well. Should have probably made a more prominent mention though :).

[1] https://smallstep.com/blog/use-ssh-certificates/

[2] https://smallstep.com/blog/diy-single-sign-on-for-ssh/

[3] https://smallstep.com/blog/smallstep-ssh/

Seems the ideal solution would be to trigger push-to-allow for signing requests that come in via agent forwarding, but not for local requests. I’ve been thinking about a reliable & secure way to do this. A modified OpenSSH client could enable this pretty easily by simply indicating to the agent where the request originated. But that’d require changes to both OpenSSH and to the agent protocol.

That works for bastions, but not other use cases. One common one is pushing & pulling from git on a remote dev box. Or if you want to SCP something between remote machines without having to pull it down locally. Agent forwarding is the best way to do this (outside of possibly generating an ephemeral key pair and copying public keys around for a single use then destroying them... but ain’t nobody gonna do that).

So you’re saying SCP will tab complete through the control master socket? Neat.

Check out our single sign-on for SSH stuff at smallstep (where I work). Either in open source[1] or our product[2]. The hassle of 2FAing all the time is one of the big reasons I love single sign-on for SSH. Basically, you do 2FA when you're pushed through single sign-on. But then you're issued a short-lived certificate that gets put in your ssh-agent. You only need to 2FA to get a certificate. So you can tune 2FA challenge frequency based on certificate lifetime.

It's sort of like a browser session where you "login" and then you can browse until your cookie expires. Here you "login" and you can SSH until your certificate expires. So you have strong authentication, but you're only asked to do it periodically.

[1] https://smallstep.com/blog/diy-single-sign-on-for-ssh/

[2] https://smallstep.com/sso-ssh/

Last time I launched a new website it took less than a month for someone to let us know that we'd forgotten to configure a AAAA and our site was inaccessible for them. And that was at new website traffic volume.

So yea, GitHub definitely knows about AAAA records and has intentionally decided not to have one. The question is: why? They must have a reason. Maybe even a good one. I'm curious.

Nice. `Match exec` is one of my favorite things. It's too bad the command being passed to `ssh` (if there is one) isn't available as a `TOKEN` (as far as I can tell). That would put a bow on everything.

I've always wondered what this is useful for. So it's purely for performance?

Why is it faster to establish the connection? Does it re-use the authentication from the existing ControlMaster too, so you skip the handshake? Seems like it could be dangerous if you're not careful. I guess that's why you can configure it to use `ssh-askpass` for conformation (which, btw, is missing on macOS these days).

The other neat-looking directive that I've never tried is `ProxyUseFdpass`, which tells OpenSSH to expect a file descriptor back from your `ProxyCommand` instead of using stdin/stdout. I'm not sure why it exists, but it feels like it could be a performance optimization. Particularly for `scp`. But I've never actually run into a performance problem using `ProxyCommand` so shrug?

Disclosure: not OP, but work at smallstep

Smallstep has a product[1] that's a lot like gravitational teleport. That's how we got deep enough into SSH to write this post. Teleport isn't bad. The two biggest differentiators are probably:

- Smallstep SSH is hosted (with HSM-backed private keys)

- Smallstep SSH does user & group syncing from your identity provider (i.e., you don't need to adduser / deluser folks anymore) so you don't need to do any user or key management on servers at all

We're also doing everything using standard OpenSSH, whereas teleport replaces your SSH client & server (or at least it used to, skimming their docs it looks like that might be changing). Authentication is via OAuth OIDC (single sign-on), user & group sync is via SCIM, plus PAM & NSS on the host side. So it's all pretty standard stuff.

Finally, Smallstep SSH is built on our open source toolchain, step[2] and step-ca[3]. Actually, if you want something completely free that does all of this you can just use those tools and do something like gravitational yourself. We have a blog post[4] explaining how.

This product is only a couple weeks old, so feedback is very welcome!

[1] https://smallstep.com/sso-ssh/

[2] https://github.com/smallstep/cli

[3] https://github.com/smallstep/certificates

[4] https://smallstep.com/blog/diy-single-sign-on-for-ssh/

We’re hiring at smallstep ;D

That `tmux` bit is clever. Wonder if you could do that in a `ForceCommand` or something like that so you don’t need to type that part either?

You can also use the `ProxyJump` directive in your `~/.ssh/config`, which is the same as `-J` on the command line. So, for example:

    Host host_final
        ProxyJump user1@host1

will do the same thing as `-J user1@host`, but will allow you to just type:

    ssh user_final@host_final

If you're using an older SSH you can do this with a `ProxyCommand` (requires netcat on the jump box, but that's pretty standard):

    Host host_final
        ProxyCommand ssh user1@host1 nc %h %p

There are a bunch of variations on this technique, but these are the most common configs. Super easy transparent bastioning.

You can get really fancy with this stuff, particularly with `ProxyCommand`. We use it to trigger auto-login for our "Single sign-on for SSH" product at smallstep. When you have a `ProxyCommand` configured, instead of opening its own socket, OpenSSH just execs your proxy command and expects stdin & stdout to end up connected to a socket to the remote server. It doesn't care how that happens or what else happens before you get there. So we (ab)use this as a hook to check if you have a valid SSH certificate in your `ssh-agent` and, if you don't, trigger a single sign-on flow. It's nifty.

If you've never read the man pages for `ssh_config` and `sshd_config`, I highly recommend it. It's not that long and there's a lot of good stuff in there.