In Defense of Mutual TLS
notion.so
notion.so
First: TLS (and mTLS) create secure channels. A channel bears many requests and responses. In many designs, a single channel will bear requests for many different users. The job of a textbook secure channel is to prevent an attacker with control of the underlying network from tampering with requests and responses. Binding identities and authorization claims to requests is a different job. The textbook solution to that problem is authenticated requests, not authenticated secure channels. I wouldn't have used the word "layering violation", which I would contend Is Not A Thing, but the point MacCárthaigh makes is important. So: when MacCárthaigh says SQLI and Request Smuggling are "still things", he's not saying that mTLS introduces SQLI, but rather that authenticating at the level of secure channels means that a request smuggling attack is almost automatically a game-over for your application, because you aren't authenticating the requests independently of the channel. Even Basic Auth can potentially avoid that problem, if an attacker has to know a secret to slip into a request in order to forge a request on a compromised channel.
Second: TLS was designed for the WebPKI, and that's its most important application. A consequence of this is that a lot of TLS software is designed for WebPKI threat models. Like everything else the IETF produces, TLS has features that support a lot of other applications, like using heartbeat messages to debug remote processes. But what matters in the real world is the installed base of TLS software, and that installed base does not have sane defaults and safe ergonomics for mTLS. You have to be extra careful rolling it out that way.
Third: TLS is built on X.509, and X.509 is extremely complex and error prone. This complexity is deceptive, because we've spent 20 years filing down the complexity of X.509 in the WebPKI deployment mode, and developers mostly don't have to care about anything except the filenames for their certificates and keys. That's out the window with mTLS, where peers actually have to crack open certificates and look at them to accomplish authn and authz tasks.
Fourth: Revocation is a debacle. It's a debacle everywhere, but TLS complexifies it, because it has to work at Internet scale. If you're managing a fleet for even a large, complex application, you have recourse to designs that make revocation easier to handle.
Fifth: Fleet-wide credential rotation is much harder in mTLS than it is in simpler systems, where the root of trust might just be a simple secret you can quickly load on to a given machine and tabletop how to do a fleetwide deployment in a quick meeting. You can add mechanism to automate certificate issuance, but those systems have to be resilient to the total loss of the root of trust, which is something the TLS ecosystem is not generally good at; the WebPKI could suffer a total loss of all trust roots if we broke RSA, but not so much if we just broke OpenSSL.
Sixth: The standard modern solution to revocation, short-lifetime certificates, also relies on secure clocks. Breaking clocks Internet-wide might be difficult enough to be outside a reasonable threat model, but breaking clocks in a single data center is not.
Seventh: To make short-expiry work in a setting where a compromised certificate isn't just a potential coffee shop MITM for a subset of your users but instead a game-over compromise for the entire application, you might need lifetimes so short they're hard to operationalize without outages. Part of what we like about public key authentication schemes is that they don't require continuous access to a central trusted authority just to keep the system up and running!
Eighth: Serious applications want queryable audit trails for inter-service RPCs, which means having ready, often indexed access to principals and roles asserted by credentials. With expansive mTLS designs, this stuff is buried in X.509 certificates, which creates friction to getting these features built; even applications built on simple bearer tokens often do a bad job of getting this right, and adding friction just makes it harder.
Finally: If you're authenticating and authorizing requests, you want a consistent, preferably rigid structure for expressing claims. X.509 provides you nothing here you'd ever want to use; instead, you're going to slip comma-separated lists into string fields, and then build ad hoc tools to grovel the information you need out of them. You can do this consistently if you (a) build everything in the same language and (b) get everyone to use the same authentication and authorization code paths (as a software security auditor: good luck with that), but you're boned as soon as your mostly-Python application introduces a Rust component.
I could rebut Malone's post point-by-point (for instance: totally unclear to me how a hello-world example of the most simplified APIs a library provides for handling certificates addressing the huge complexity of X.509), but I'm not yet sure I need to; I think a fuller recitation of what MacCárthaigh was trying to say does a better job.
I will say that I'm actually not an mTLS opponent. As I recall, Colm's thread was prompted in part by a draft of my "Child's Garden of Inter-Service Authentication Schemes" post, which was mostly motivated by hatred of JWT and originally spoke warmly about mTLS. And I still like mTLS just fine --- for simple network topologies and trust models that are flat and uniform at the network layer. By all means, use mTLS to make sure only real apps in your environment can talk to Consul, and to make it harder to SSRF things. But be wary about asking it to do more than it's good at.
I await Colm's savaging of my attempt to say in 489479837 words what his short Twitter thread did a better job of communicating.
Not too long into my tenure of being an active maintainer of Apache httpd I somehow ended up being the X509 person. I'd worked on some of the crypto code in Apache-SSL, which was maintained by non-US people only at the time, and X509 sort of fell into that bucket. I also got on well with Dirk-Willem van Gulik who wrote a lot of the original X509 code, and later he ended up hiring me, but that's a different story.
As part of being an "Apache x509 person" I got a consultancy gig to audit an "Enterprise" company's mTLS setup. On the first day, based on nothing more than tailing logs, it became clear that some people had access to things they really shouldn't. The CTOs Executive Assistant, who was part of the group trying out mTLS, actually had god level access to everything. EAs are gods and should be worshiped, they make companies run, but this level of access was not intentional. A few hours of root causing and I figured out that the EA had god-level access to everything just because the string "admin" was in their job title. Access was being granted based on a greedy PCRE regex match on a string. It wasn't even in the right field in the certificate, but the string matching didn't care. This is what x.509 does to your brain.
Similar gigs would find more problems; to an approximation, no one had working revocation, more than once I found setups that didn't even require client certificates, but no one had noticed. Over time I just got the impression that x509 was commonly set up by folks who were blinded by a cryptographic luster but wouldn't follow through. There's been a general absence of professionally built solutions that get the details right.
Literally nobody likes x509. Is anyone seriously working on an alternative? I think there’s room for a more modern, simpler cert format.
Sort of relatedly, I actually really like JWK for representing keys.
Do you think a professionally built solution that gets the details right is possible?
It doesn't have to be AWS, but I do think that solutions in this space really have to be operational services, and not just technologies or standards. It has to be someone's full time job to look at this stuff and make sure it's working and secure, and that's just very hard to match. If I quit my job tomorrow and was building a startup, I'd use AWS API Gateway's SIGv4 and policy support and leave it at that.
I’ve always thought of AWS IAM as: great for my-service-to-cloud-service, not so great for my-service-to-my-service.
I haven’t looked at AWS IAM in a while. I’m gonna take another look now :)
Reading it over, I think we mostly agree on the facts. It's easy to do mTLS and x509 wrong. The question, then, is what's easier / more secure: doing mTLS/x509 right or doing something else? I think that's somewhat subjective: it depends on your requirements, your environment, and your skillset.
One point that I'd like to reiterate is this: if you want a consistent cryptographic solution that works everywhere, TLS is pretty much your only choice. You could use something else for client authentication, but you probably still need TLS.
As a strawman, here's a sketch of how I'd recommend doing TLS in a microservice system. I consider this "right" for most garden-variety microservices-in-cloud scenarios and don't think it's particularly hard to do. Most of this is already implemented in https://github.com/smallstep/certificates:
* Deploy the root cert via automation (so it's quickly rotatable) and/or keep it in a managed HSM/KMS. You might harden root rotation a bit by signing your new root with your old root. But, generally, trust config management or container orchestration to push root(s) (you already trust it to push code and secrets). Root rotation (and, thus, bulk revocation) is now as fast as secret rotation (secrets are generally pushed the same way).
* Issue short-lived certificates per logical entity. If it gets a box and a name in your architecture diagram, it should get an identity and each instance should get a certificate. Use domain names and email addresses that you control for names. Keep certs simple: one SAN. Certificates bind a name to a public key. That's it.
* Automate certificate issuance. ACME can work for this, but there are other options (single-use tokens issued by config management, cloud-managed instance identity documents or service accounts, an existing device certificate issued by a manufacturer, etc.)
* Automate certificate renewal. A simple mTLS HTTPS request works for this. This is easy to implement and easy to scale out with multiple intermediates. "Revoking" a certificate just marks it as "not renewable". To reduce risk of outage, in this architecture, it's safe to renew an expired certificate as long as it's not revoked (ACME-STAR basically does this, but it's push instead of pull).
* If you really need active revocation, fine. One good solution is to push CRL to a cloud storage bucket. Short-lived certs will keep your CRLs small. If you need to do a mass rotation, rotate roots (push new root, wait for rotation, pull old root).
* Use secure NTP for time.
* Index issued certificates. CT (trillian) is cool if you want to be fancy. Your existing database or SIEM also works. zcertificate can parse x509 and output a JSON representation of a certificate that you can map to something like an Elastic Search schema: https://github.com/zmap/zcertificate
On your first point: I understand that in theory an attacker could slip a request across a secure channel, and binding authentication to a request could in theory prevent that. I don't understand how that's likely to happen in the context I'm thinking of here. Which may be different than the context you're thinking of. So let me clarify.
Suppose I have `<end-user> -> <service-a> -> <service-b> -> <database>`. Let's focus on `<service-a> -> <service-b>`. I don't see how using end-to-end mTLS, terminating in `<service-a>` and `<service-b>` application code, would be any more vulnerable to this variety of attack than an HTTP Basic header like `Authorization: Basic base64(service-a:password)`. Surely, the logic in `<service-a>` is simply "insert HTTP Basic header into requests on their way out to `<service-b>`". It doesn't matter if we're authenticating the request or the channel. If you're able to smuggle something malicious into that request, it's gonna get sent over to `<service-b>` with proper authentication attached.
Are we talking past one another? Are you trying to make `<end-user>`'s authenticated identity carry through `<service-a>` to `<service-b>`? If that's the case, then yes: I see what you're saying and you shouldn't use mTLS for that. I'm not sure if there's a term-of-art here, but I call this "end user identity propagation". You need something like a top-of-stack ticket service (a bearer token) for that. Or, better yet, macaroons. I consider those two separate things, though. mTLS is for authenticating your immediate peer. For end-user identity propagation mTLS is a poor choice.
On your final point: you could, in theory, express claims in x509. I'm sure you're aware, but it's been tried before (e.g., SPKI/SDSI). However, I agree that, unless you really know what you're doing, x509 is too complicated for that. Don't do it. You'll likely screw it up. If you're parsing x509 and ASN.1, you're doing it wrong. If you're processing strings that you've extracted from a certificate, and you're not in the habit of writing your own formal languages, you're definitely doing it wrong. Just put a flat name in a SAN. The only thing you should ever need to do with that string is an exact string comparison. If you need to know roles or groups or some other metadata look them up in a database.
(Or use macaroons)
I'll stipulate to mTLS resolving client-identity and server-identity. But the client is making requests of the server that pertain to a user. The certificate doesn't (and can't) attest to a user.
In most designs I've actually evaluated, what ends up happening is that the server just trusts the user provided by the client, and the client tries hard not to ask for things for the wrong user. But beyond whether the client has authorized every code path that generates a request, you have an additional problem here, because even after the client authorizes a path that generates a request, a bug in the client can give an attacker influence over the request itself. The server has no way of distinguishing between a corrupted request and a real request, even though you're relying on a secure channel.
Authenticated requests mitigate this problem: an attacker might be able to corrupt a request, but it's not enough to corrupt it; you need to know whatever secret authenticates the request itself.
To do this right, you now need two authentication schemes in play: one for the secure channel, and one for the requests themselves. But if you can reliably authenticate requests, why are we rigorously authenticating the secure channel? We're spending complexity chits to buy only marginal extra security.
I think it can make some sense to mesh up services with mTLS as a "you must be this high to get on the ride" mechanism. But since that's all it's doing, we don't need a lot of complicated mechanism to give precisely the right certificates to services, because even if something gets screwed up, request authentication, not secure channel authentication, should be what's protecting your application.
Especially when we think about short-lived certificates and ACME and rollover: what is this really buying us in a strong system with request authentication? A WebPKI ACME certificate will live for 90 days. If certificates matter to our design, we can't let a compromised cert live for 90 days. Our certificate lifespans have to be much shorter --- the duration in which we'd be OK having a compromised credential still viable for an attacker. Remember the differing consequences between a WebPKI certificate compromise and an internal credential compromise: one is a second-order flaw that lets attackers with some other vulnerability MITM a subset of users; the other is a game-over flaw. How long would you let a compromised developer prod SSH key live?
So now we're talking about very short-lived certificates, and you can see, this is a design that spends all the complexity chits asymmetric encryption costs, but is asymptotically approaching the operational resiliency of Kerberos. Yikes.
The WebPKI has to assume certificates might be compromised because there are 18 zillion different ways people might mishandle a certificate. But that's not at all true for a data center deployment of a microservice ensemble! Certificate keys are as secure in our system as the microservice itself is. If you lose one to an attacker, you lost the microservice too --- even if you rotate the cred, whatever flaw gave the attacker that certificate is just going to give the attacker the next one, too. If you stop trusting a certificate (and, by extension, the instance it was resident on), you zap the whole instance, revoke, and root-cause the flaw that caused the problem. Rotation just isn't winning you much.
At bottom the issue here is that the WebPKI has a very particular threat model, and it's a shitty threat model, and we have built lots of tools for that threat model, and many of them are by necessity shitty because we live in a fallen world. But your microservice ensemble wasn't born with original sin! You don't need to inherit the complexities (e.g. X.509) and limitations (e.g. X.509 revocation) that the WebPKI has to deal with. Which is a reason some people get itchy when we talk about importing WebPKI technology to things like K8s.
I tend to be pretty chill about little islands of mTLS, like "all the Consul clients need a cert". That, to me, solves a practical problem. I am way less chill about attempts to create coherent PKI namespaces for all the components of an app, tied together with mTLS.
People should use Macaroons!
First, agreed. Certificates can't and shouldn't attest to the user's identity. You need another mechanism for that.
If you're securing requests, then securing the channel might be redundant. However, I do want authenticated service identity (not just user identity). The shape of policy I want to enforce is: "<service x> should be allowed to make <rpc> to <service y> in the context of responding to <request> from <user z>". I need the authenticated identity of both the service and the user to do that. Either that, or the authorization could take place higher up in the stack and I need a capability token passed through. Even with a capability token, it's nice to know the authenticated identity of the calling service for other purposes, like audit. Not to beat a dead horse, but I realize that macaroons satisfy all of these requirements :).
I also want confidentiality. Macaroons don't do this. This is an important point that I want to make more strongly this time, because I feel like it's getting lost in the noise. Authentication isn't the only thing people are using TLS for.
I appreciate your perespective. I think the best way to respond is to actually answer @halvarflake's question: what's the threat model that justifies service-to-service mutual TLS?
One answer is: that's the wrong question. Many people use mTLS for flexibility, not security.
Most microservice systems are born as a bunch of web services thrown in a VPC (or, more recently, a kubernetes). They can all talk to each other. Some of these systems are successful, and eventually outgrow their perimeter. Sub-components need to be able to communicate across untrusted channels (e.g., the web). TLS can give you a "cryptographic VPN" to do this.
There are a lot of VPN-y solutions with a lot of pros and cons. TLS isn't always the right way. The big pro for TLS, however, is that it works everywhere. Even if you don't control the host stack (e.g., functions as a service) or if you're on a constrained device, etc. TLS is also baked into most infrastructure: proxies, queues, databases, etc. You can easily terminate TLS easily at your perimeter if that's what you want. Alternatively, you can connect directly from a service in one perimeter to another service or to a piece of infrastructure like a database in another, without a proxy. Again, TLS is ubiquitous and it's flexible.
At this point I think we're at your "you must be this high to get on the ride" model. Why go further? This is where it makes sense to start talking about threat model.
First... I already mentioned this, but, once again: confidentiality. I think this is less controversial, and it doesn't require client certs. So I'll not elaborate further except to say that people need it.
The interesting topic is: service authentication. Why use client certificates for service authentication? I contend that service-to-service mTLS plus a bit of simple, coarse authorization actually does address a number of common security threats. Specifically:
* Malicious insiders: if you're using microservices and practicing DevOps and agile and have engineers on-call you probably have a lot of people and a lot of code that can get inside your perimeter. Coarse segmentation of service-to-service interactions can help here. An engineer who has the ability to `ssh` or `kubectl exec` or tunnel into your perimeter won't immediately have access to every sensitive service you run.
* Lateral movement: very similar to the above, but this time a service is popped or an engineer's laptop is compromised. Service authentication helps control the blast radius if someone gets inside your perimeter.
* Credential leaks: secrets are hard to manage. They get committed to repos, accidentally logged, dumped in error messages and served to end users, pasted in to slack channels, walk out the door with dismissed employees, etc. Using certificates with a relatively short lifetime gives you assurance that compromised credentials will be eventually rotate once an attack vector is closed. I contend that it's actually a lot easier to set up client certificates that automatically rotate than it is to rotate secrets for all of your proxies, databases, services, etc. An asymmetric crypto key is also inherently less likely to leak to logging infrastructure or end users than a password or bearer token since it's not actually in any requests.
Request authentication could mitigate these threats, too. And, again, carrying an authenticated end-user identity through the stack to downstream services is valuable. But, in a microservice system, I'm less worried about code lying about the end-user it's acting on behalf of than I am about malicious humans. If I already have TLS for "my VPN", authenticating clients and coarsely authorizing service-to-service interactions starts looking pretty attractive.
Regarding compromised keys, certificate lifetimes, and revocation... that's all important, but less critical to the threat model than it may appear. If a key is compromised I'm still much better off than I was without my coarse segmentation: the only stuff you can access is the stuff that the service is authorized to access. If an engineer exfiltrates a key to "test something", that key will eventually get rotated, even if I never know it happened. Same with an attacker exfiling a key: they have to maintain a presence on compromised systems to exfil new keys as they rotate. (FYI: by default, the smallstep toolchain issues certs for 24 hours. Personally, I'd push that down to a few minutes.)
It's also worth remembering that we still have our perimeter (or perimeters). If we find an attacker, we can lock them out at the perimeter instead of revoking a certificate. Even if you can TLS through the perimeter, we only need to enforce revocation at stuff directly exposed to the internet (e.g., in a proxy, not necessarily in every service). Active revocation is nice. But it's not necessarily critical. As another example, since you mentioned SSH: smallstep's SSH product uses certificates without active revocation. We revoke access in policy, by pushing ACLs to hosts and locking users out via NSS and PAM. The certificate may still prove I'm "mike", but mike doesn't have access anymore.
Why not do all of this host-to-host instead of service-to-service? Well, depending on your requirements, that may make sense. Certainly in kubernetes-land the line between "a host" and "a service" are blurring. TLS still has a few advantages:
* It works in places where you can't control the host (e.g., functions): it's a consistent mechanism that works everywhere
* It makes applications more self-contained and, therefore, more portable
* It's logically closer to what you want: you're trying to authenticate services not hosts
* There's more deployment flexibility: you can terminate in a sidecar or in the application
* It's easier to pass authenticated identity information into the application
* It works for authenticating to infrastructure: proxies, databases, queues, etc (again, it's a consistent mechanism that works everywhere)
Fin.
(Are we using macaroons yet?)
Another sleight of hand I see here is the ambiguity between network authentication, service authentication, and user authentication. When it's convenient for the argument, it seems, mTLS is just a way to ensure both ends of a network connection are part of the super best friends club. But later, when it's helpful to argue that mTLS is "flexible", certs carry credentials. Maybe even JWTs! There's a subtext to the "be wary of mTLS" argument, and it's that unified site-wide PKIs don't work well in practice.
Finally, you've lost me completely with this "revocation is less critical to the threat model than you think it is" stuff. The step-ca website is even explicit about this distinction you're trying to draw between "active revocation" and "passive revocation" (short-lifetime certificates). No. That doesn't work in internal application environments. step-ca defaults to 24-hour certificates. Have you considered how weird that is? 24 hours is short enough that your CA needs to be highly available --- ask around about production outages that have been caused by certificates expiring at much longer lifetimes --- but long enough that you can't possibly "wait out" a compromised cert.
I feel like something that is happening here is that WebPKI-style mTLS advocates think that a server cert and a client cert are two different instances of the same thing. They are not. They are wildly different entities. A server certificate prevents a MITM active attacker from intercepting a TLS connection. That's a rare attack, and, more importantly, a second-order attack. A client certificate is an access token for the application. Lose a prod-viable SSH key and then tell me I should wait 7 hours for it to expire.
The reality is that every application that deploys "passive revocation" is ultimately going to have to implement "active revocation", and, if they follow the advice on the step-ca website, they're going to have to do it on no notice, and in the most disruptive way possible, because when an internal access credential is lost, it has to be shut off immediately.
I'm also not arguing that some active mechanism to immediately remove attacker access is unimportant. Of course it is. I'm arguing that active certificate revocation is not the only way to achieve that. I gave two examples: remove access at the perimeter or push new policy to revoke access for the compromised entity. Most microservice systems already have tools here: they're typically adding mTLS on top of a perimeter model with decent privileged access management.
This is a bit of an asymmetric argument, in any case. I have all of my cards on the table. The opposition isn't really proposing any concrete alternative, so it's hard to make a meaningful comparison. If you want active revocation, implement active revocation. It's not easy, but OCSP and CRL exist and they do work. I even suggested a way to do it well: push a CRL to a cloud bucket, which is scalable and highly available. You can call that "cargo culting all of the idiosyncracies of the WebPKI", but I still haven't heard a concrete alternative. So where does that leave us?
The only concrete thing that's been proposed... that I had to propose myself, since no one else would even answer this very simple question, is macaroons. Which I've said repeatedly I think are a good idea. However, it's abundantly clear to me why people, in practice, are choosing client certs over macaroons. Macaroons can't layer into an existing system the way that mTLS can, and if all you want is "blast containment" you can get that with a fairly simple client certificate setup and a bit of coarse authorization.
Regarding CA availability: if you can't remediate an availability issue with a shared-nothing web service in 8 hours (the downtime you'd need for a cert to expire in a default step-ca setup) then you've got bigger problems than certificate management. The 24 hour default is long enough for people to remediate an outage and short enough to get people operationally comfortable with the idea of short-lived certificates. If I were to use shared secrets instead of asymmetric keys, and I created a system that rotated all of my service-to-service shared secrets automatically every 24 hours would you say that's a bad idea?
Secondly:
I'm also not arguing that some active mechanism to immediately remove attacker access is unimportant. Of course it is. I'm arguing that active certificate revocation is not the only way to achieve that.
An "active mechanism to remove attacker access" is revocation. That's literally what the word means.
Re: revocation... the point is that you can revoke attacker access without revoking a certificate.
Or, if you want, you can revoke certificates. I’m not totally opposed to that. CRL is finicky, but architecturally it works well with short-lived certs and it’s not very different than secret rotation. If you can push new secrets, you can push a CRL. In fact, if you can push new secrets, you can also push new roots.
There are many people who use mTLS. Along with consistent service logging, it’s one of the main reasons people use service meshes. It’s also very common in IoT.
If you find someone malicious in your network, you lock them out of your network. You don’t need to revoke a certificate to do this.
OR, you can do CRL. This isn’t jazz hands, it’s a choice. As you push mTLS harder, and need things like active revocation to satisfy your threat model, it’s worth reconsidering whether mTLS is the right choice. Sometimes it is, sometimes it’s not. Sometimes you don’t have a feasible alternative.
You can't "lock someone out of your network". That's handwaving. If you can do that reliably, you don't need any other security controls. Just lock all the bad people out!
The problem with mTLS in configurations where you rely on it for authentication is, if an attacker manages to obtain a client certificate key, it no longer matters if you nuke the machine from orbit, because they have the keypair! The keypair still works! They can use that keypair from any other place on the network they have access to (if there's no such place, you just defined away the need for mTLS).
Which is why you need active revocation if you're going to rely on mTLS: when you lose confidence in your sole custody of a client keypair, you need to actively revoke that keypair, immediately. You can't just wait 7 hours for the step-ca default key lifetime to expire!
"Or, you can do CRL" is literally active revocation. Your project has a prominent call-out about how most people don't need to do this. But that's literally the opposite of the truth. It's true in the WebPKI, but I think you've become confused about the difference between WebPKI server certs and internal client certificates. Telling people they should deploy lots of mTLS but not worry about revocation is malpractice.
What's going to happen to people in practice is that they're going to have to re-key their entire fleet any time there's any question about the integrity of a service. Or, more likely: they won't, because the mTLS deployment mode you're encouraging creates so much goddamn friction to protecting a key that people will roll the dice with the safety of their users rather than confront the fact that the correct engineering solution is an outage-inducing all-hands-on-deck rekeying.
What blows my mind about this is, at 24-hour expiry, in a medium-sized application, you're going to have machines needing to refresh keys practically every hour of the day; your CA will need to be available 24/7. At that point, you've basically reinvented Kerberos. Ops teams fucking hate Kerberos! And for all that effort, you still face fleetwide updates any time something sketchy happens anywhere.
I like mTLS for things like "we set up a Consul cluster; let's make sure just the machines that use Consul can reach it". It works fine for that. I don't think it's a good idea to take it much further.
If you have a perimeter with well-defined entry points (e.g., a bastion for HTTPS and SSH) then you can lock people out at those entry points. Mutual TLS is defense-in-depth, and how you implement least authority for services inside the perimeter.
I know that CRL is literally active revocation. If you want active certificate revocation you can literally do active certificate revocation. Whether you need it depends on your threat model. And it’s not actually that hard in many scenarios. It is harder than not doing active revocation, and if you find yourself needing it to satisfy your threat model you probably should consider other alternatives for client authentication.
I’m not telling people they should use lots of mTLS but not worry about revocation. Any content I have written that discusses short-lived certificates is very careful to point out the fact that a compromised certificate will be considered valid until it is revoked. Most people who are using our open source toolchain are actually using it for internal server auth TLS, for narrow use cases like issuing certificates for database authentication or VPNs, or for mTLS to satisfy the sort of threat model I’ve described here.
We haven’t implemented anything around active revocation yet. But it is planned, because it is necessary in many scenarios.
If you’re worried about keeping a simple web service running 24 hours a day then you shouldn’t use our stuff. If you’re far enough along to need our stuff, you’ve figured out how to keep a web service running... you have to keep your own web services running 24 hours a day otherwise your microservice system is already going to have availability issues.
You don’t need revocation because the credential is useless outside of the perimeter. If you find a malicious insider or a compromised service, you fix the service or lock the insider out of the perimeter. You’re glad that you had client certificates in place as it reduced the scope of the compromise. It would be nice if you could actively revoke any compromised credentials, but it’s not critical because you’ve locked out the offending user and/or fixed the compromised service.
I think we're probably at an impasse.
Credential rotation is good security hygiene. To suggest otherwise is malpractice. Our toolchain makes certificate rotation trivially easy. Why not rotate frequently?
Hopefully the threat model stuff made sense. It still feels like you actively want to disagree with me, and I’m still not sure why. But I agree that this is starting to feel unproductive.
I do appreciate the discussion. I understand your position on client certs better now. Your concerns are valid.
Maybe one day we can discuss over beers or something. It feels like that would be the right atmosphere.
Furthermore, it’s not arbitrary. Credentials leak and services come and go. Having active keys around that aren’t in use is worse than not having them around. If someone accidentally commits a key to a GitHub repo or something, it’s nice to know that key will only be useful for a little while.
If you still want to rotate less frequently, change the default.
This really has very little to do with the topic at hand, so I’m not sure why we’re debating it. Do you want me to change the default certificate lifetime in step-ca? What do you think it should be?