Using Let's Encrypt for Internal Servers (2018)
blog.heckel.xyz
blog.heckel.xyz
My company (Datto [1]; we're hiring, see [2]) sells backup appliances which our customers place in their company network (much like you would place a router in your own network). Since we do not control anything other than our appliance inside our customers' networks, we went this route to provide a secure web interface to our appliance. And since the 65k servers/appliances (now more like 80k) are located in tens of thousands of different networks, leaking the internal IP isn't bad at all -- especially since there is no way to correlate them with the customer.
For normal "internal servers" within a company, I'd probably recommend using a wildcard cert and an internal DNS server.
Also: Yeyy, I'm on HN!
[2] https://news.ycombinator.com/item?id=19071727 or https://www.datto.com/careers/
See:
https://github.com/cloudflare/cfssl
You seem to be making many assumptions here, which some may not agree with.
Maybe not directly - and I say that as a maybe, since I'm not so sure of this premise - but you're leaking a bunch of data this way that can be used by attackers that have some level of information or access to get additional information about the network, the services running on it, etc. The idea that this "isn't bad at all" is pretty much patently false. "Not too bad in most situations" is probably a bit more accurate.
>For normal "internal servers" within a company, I'd probably recommend using a wildcard cert and an internal DNS server.
Why? For internal servers within the company you can run your own PKI and CA and use an internal cert, giving you much more flexibility and control over things than just using a wildcart cert and internal DNS server would, without much increase in effort.
>We went this route to provide a secure web interface to our appliance.
The far better option is to give the customer a way to provide their own cert. Or, since you are providing the appliance, have them trust your CA on the machines that are administrating the appliance.
I'll be frank: Your choices here and general suggestions around DNS would worry me if I was in the market for your appliance. I would not find it at all acceptable that internal details of my network are leaking out, no matter how convinced you are that it's not bad at all.
Running an internal CA those days terrifies me, its a hell of a responsibility to keep that secure. People check medical records, financial data, and god knows what else on their work devices, and if your CA is compromised that can all be eavesdropped on. Either by an attacker or disgruntled insider. No thanks.
So if Example Inc puts their cert on every employee's machine, then a hacker or rogue insider could issue a cert for paypal.com and MITM every employee.
Starting an internal CA introduces the risk of this bad thing happening.
Better not have Active Directory then, because it has a built-in CA used by every Windows client bound to the domain.
Technically, X.509 allows for constraining things [1], but from a practical perspective [2] it's not really implemented.
[1] https://tools.ietf.org/html/rfc5280#section-4.2.1.10 [2] https://security.stackexchange.com/questions/31376/
Get an HSM - they're pretty cheap these days - and store the CA's private key in the HSM.
Or an offline CA, if you don't want to deal with the HSM.
Or, even better, do both.
If you're an absolute tightarse it's even possible to use a smartcard/yubikey as a poor-mans HSM
So absolutely, yes, you can run your CA reasonably today. But in practice, outside very large and competent enterprise shops (i.e., your typical business) - no. The typical business I bump into struggles with internal DNS and DHCP - the skills just aren't there.
And wildcard certificates imply sharing a private key around which is terrible advice
If you're a business spend the money on certificates, if the business pushes back ask them "how much is your data worth? How much will reputational harm cost us?"
Lots of people suggest rolling your own CA but I bet most of the internal CAs out there are terribly secured and not audited.
While I agree that it probably shouldn't matter it could help an attacker gather knowledge about your internal network. Perhaps it's of interest that you use a specific monitoring tool or mail server internally - that could make phishing emails easier to craft.
Edit: Or do you mean as part of the challenge-response mechanism? Yeah, that would have to be public.
Besides possibly being a function of a provider's API; DNS server security policies can be used to limit updates to certain domains and/or record types based on preshared key. Since the DNS-01 challenge only needs to make a TXT record with a predetermined name you can configure a zone like so (using BIND syntax as an example):
key "example-key" {
algorithm hmac-sha512;
secret <KEY_HERE>;
};
zone {
...
update-policy {
grant "example-key" name _acme-challenge.example.com TXT;
};
};
However how long before the default option is via https only.
You could use a private CA or official wildcard certs.
If you're hostnames are sensitive then Web PKI is not the correct solution, like you have identified. At that point you should be running an internal PKI and be responsible for running/maintaining to whatever standards you deem fit.
Making Web PKI less transparent to fit your use case is not the answer.
> CT is a bank alarm that goes off after the bank's been robbed
> There are better solutions
How do you propose to prevent invalid certificate issuance? In particular, how do you propose to observe whether any CA accepted by browsers has issued a certificate to someone other than you for a domain you control, if you presume that there are malicious actors willing and able to compromise certificate authorities? Your threat model is nation-states. Assume someone else is willing to put up the cash, what's your solution that works strictly better than certificate transparency, and in particular (since you called attention to it) prevents invalid certificate issuance rather than just logging it for subsequent audit (and CA trust revocation)?
Things like certlint have come about to help prevent misissuance, but I would wager that most CAs have not added it to their issuance pipeline.
I agree that CT is not the solution and ideally it would not be necessary, however, the number of issues found and still being discovered justifies it. Trusting CAs to just issue proper certificates has been a failed policy.
Sure, someone can see you have internal servers under * .homelab.mydomain.com, but they aren't going to know what specific hostnames exist there.
It effectively automates the process that's described in the article. Once you have it set up (you need a few DNS entries and a tiny Go server running somewhere), using it is as simple as issuing an HTTP API call to your alley-oop server with the LAN IP you have, and the DynDNS domain you want associated with it, and getting a valid cert in return. You're then ready to spin up your HTTPS server with it, and start serving LAN traffic with a valid cert.
Both Caddy and CertMagic support nearly 3 dozen DNS providers, and adding your own is relatively easy as well.
CertMagic: https://github.com/mholt/certmagic
Caddy: https://caddyserver.com
In most large companies, issuing certificates remains a manual process. Generate a CSR, create a ticket, wait for response, download certificate, install on host. ACME seems like it should be the standard API for fixing this.
No ACME of course, but considering the MSFT certificate management protocol pre-dates ACME by more than a decade that’s not surprising. Traditional manual issuance after CR submission workflow is also supported for any OS.
EDIT: Based on the fantastic CertMagic library by mholt
A service I sell is a callcenter SAAS. It uses twilio to make outbound calls, but this won't work on internal servers. (I prefer to deploy the service as appliances that get shipped to the customers).
So this is a great thing for me, since it means I can start offering my customers the ability to make outbound calls from their appliances.
But yea maybe parent poster could consider scheme: git-internal.yourdomain.com.
> The name constraints extension, which MUST be used only in a CA certificate, indicates a name space within which all subject names in subsequent certificates in a certification path MUST be located.
That is, you get a CA certificate signed by some trusted CA; that essentially makes you a CA; the nameConstraints section restricts it to a certain set of names, so the rest of us are okay w/ you being a CA as we know you can't issue for google.com. E.g., a nameConstraint of ".mycompany.com" allows you to issue under mycompany.com.
Sadly, and this is the key bit, AFAIK, this is completely unsupported by browsers¹ and CAs. So, it's not possible to get one, and AFAIK, even if you did, it wouldn't work. I really wish this was different, since it would make the whole certificate thing considerably easier for a lot of usecases, IMO, and is a more sensible way of doing things.
[1]: https://tools.ietf.org/html/rfc5280#section-4.2.1.10
¹I think Firefox might support them, but I think that might be it.
If you control a name constrained subCA for example.com, then new leaf certificates can appear under there at your whim, but there are a bunch of rules that need to be followed for leaf certificates, so how do those get enforced? The root CA is responsible for ensuring they are.
One option is you could say OK, they just don't. So then certificates under these constrained subCAs are _worse_ than the rest of the Web PKI, why should we trust these subCAs? Clearly browsers should forcibly distrust them in that case.
Another option is, the root CA has physical control and all issuance goes through them anyway. But if you do this (and a few outfits have done it) then there's no practical benefit to the constrained subCA existing at all. It's just extra baggage.
But that's not a huge obstacle, account based issuance exists today in the corporate space, some tech person proves control every so often and as long as they keep that up to date all other account users get certs whenever they want.
It's the other rules that we don't trust you to enforce once given your own subCA.
1. Lifetime is an easy example. Having checked you own example.com the maximum longevity for a leaf certificate is 825 days. A subCA that itself only lasts 825 days would be kind of annoying (you'd need to replace it say once per year, then swap all your issuance to the new one each time), so presumably you're going to want it to last longer, whereupon nothing prevents you using it to issue yourself leaf certs that last longer even though that's forbidden.
2. Key checks are another rule for lead certificates that the root CA is supposed to check but if you have physical control how can they?
If you ask Let's Encrypt for a certificate for your Debian weak key because you forgot that the archaic Debian system you were trying to certify has a broken OpenSSL install out of the box it will just refuse. That key is no good. But how can the root be sure you run such checks every time on your own subCA ?
3. Or how about compatibility. The trust stores and the Baseline Requirements both say only to issue specific types of certificates, e.g. no SHA1 for security reason, but also no SHA3 because their software doesn't grok that yet. Or equally No putting arbitrary EKUs in certificates without explaining why. Rules like this also can't be enforced on the subCA.
Regarding 1: that's interesting. I guess I figure that part of validation should be to check that the expiry of a cert is within the expiry of the issuing CA certificate. But if this is an issue… what prevents CAs from doing this today, and if it's just "good behavior", why not codify this in the validation rules? (That is, any certificate can't expiry after the expiry of the certificate above it in the chain?)
Regarding 2 & 3… is not saying "it's the responsibility of the domain owner" not sufficient? Issuing poor keys only hurts them (and perhaps those who want to communicate with them, which that might be an issue). (I agree that this isn't a great response, and it would be better for the world as a whole if we got these checked, but is it really our job to force domain owners to do this? Also, IDK if I really trust that major CAs are doing a good job of checking these sorts of things given all the other issues we see with them.)
SHA1 is going to be ignored by user agents soon (already?), so a subCA won't issue those solely b/c they won't work. What's wrong with a subCA adopting SHA3 sooner than major CAs? (Given how long the switch off of SHA1 took … that seems like a plus?)
Better then to reject this whole "Let's give subCAs to random people to save them some work" approach altogether.
Can you trust that major CAs are doing a good job? Well, if you suspect that they aren't all the data is there to go look for yourself, please report anything untoward that you find to m.d.s.policy and/or the issuer.
For the final point - the problem is compatibility. We do not want to create myriad non-interoperable systems, that's why "the Internet" won in the first place. If not for the grave security problems with SHA1 (demonstrated collision for a relatively affordable sum of money) we'd be using that forever, even though newer hashes are nicer.
SHA3 in particular might never go anywhere. Its design (a sponge not another Merkle–Damgård construction like MD4, MD5, SHA1 and SHA2) has appealing properties such as preventing length extension attacks, but you can get many similar benefits from things like SHA512/256 (run SHA-512 but throw away bits to keep only 256 of them).
> nameConstraints in a root cert
I'm talking about nameConstraints in a non-root certificate; IDK if this has an effect on how browsers would or would not validate it. My comment is simply "last time I tried it, it didn't work." It was a few years back, so perhaps I should try it again.
It uses the let's encrypt client (certbot) and nsupdate so I can use any RFC2136 compatible DNS server, in my case, Bind. This has been running on the backup MX since I wrote it, without any hickups so far.
This can also be used for. Internal services, since you do not expose anything to the internal machine.
Set up a Lets Encrypt wildcard cert, and use it on your internal LB. Set up your internal DNS to autoresolve custom internal hostnames. For example, if you own example.com, set up internal hostnames such as test1.example.com, test2.example.com
These hostnames are internal, so they won't be resolvable ( or known about ) outside your private network. The wildcard cert combined with SSL termination at the LB means all your hosts now have ssl certs.
The solution I suggested above is for home/personal use. But if you're doing anything for production, you really shouldn't be using LB ssl termination anyway, and might even benefit from running your own internal CA service.
>"To increase this number, you have to either request a higher rate limit or get your domain added to the public suffix list (note: adding your domain here has other implications!)."
Can someone say what those other implications are? I'm not sure why the author would mention this and then fail to state what those are.
In a nutshell adding your domain there makes the subdomains completely isolated (they can't set cookies for higher level domain).
This is a good idea if you're hosting user pages as subdomains. See also PRs https://github.com/publicsuffix/list/pull/722
Is it not possible to get a root-signed intermediate CA for someone who can prove control over a domain? This would allow you to issue certificates for "xxx.internal.mydomain.com", without the need for a wildcard certificate, and without the need for using a public CA for every individual certificate?
CT would still be a problem unless the CA could be flagged to "allow non-CT certificates", and the browser ignore those requirements as they ignore CT requirements for manually installed root certs
The benefits to this is 1) there's no need to manually install a root certificate on each client device 2) Your internal domains are not reliant on an external CA
Technically, X.509 allows for constraining things, [1] but from a practical perspective [2] it's not really implemented.
[1] https://tools.ietf.org/html/rfc5280#section-4.2.1.10 [2] https://security.stackexchange.com/questions/31376/
It's really nice not to upload a challenge file for doing so.
Tutorial here: https://about.gitlab.com/2016/04/11/tutorial-securing-your-g...
Sadly, I don’t think any browsers support such a thing.
https://github.com/irtnog/certbot-formula
https://github.com/irtnog/salt-states/blob/production/certbo...
https://github.com/irtnog/salt-states/blob/production/salt/f...
There are a few bits I haven't published mostly due to laziness on my part, like how I handle certificate assignments, but it's all pretty straightforward. The hardest part has been developing the necessary configuration scripts for Windows. It takes some effort to get keying material installed in the appropriate certificate store, which you can see part of here:
https://github.com/irtnog/salt-states/blob/production/wincer...
And then there's how things that use CryptoAPI/CryptoNG reference the desired certificate by SHA-1 thumbprint, which you can see part of here:
https://github.com/irtnog/salt-states/blob/production/iis/ce...
https://github.com/irtnog/salt-states/blob/production/rd-gat...
In some cases there isn't a clean API for certificate installation:
https://github.com/irtnog/salt-states/blob/production/rdp/in...
In other cases one must run the necessary PowerShell cmdlets under user accounts with the correct privileges, which is something I haven't quite figured out (e.g., for Exchange 2007 and newer).
Sometimes the services in question require the keying material be structured in very specific and slightly odd formats (e.g., Splunk). Sometimes, I just stick a load balancer in front of it, terminate the TLS connection there, and call it a day (e.g., Tableau), perfect being the enemy of good and all.
I've given a lot of thought to the question "how would an attacker use CT logs against me", but I think the probable losses of something untoward happening due to certificate transparency is extremely small compared to attackers getting onto my networks and eavesdropping on sensitive, internal comms. I'm also not confident in my ability to run my own internal CA, both because it'd be a lot of work to secure and because the bus factor would be so damn high. At least this gives the rest of the team some incentive to tie new things into the configuration management system because they get certificates for "free". Maybe there's a better way. I don't know.
The rate limits alone seem to be a potential danger if they need to reissue new certificates for their 65,000 servers.
Even if you do have an expert available, it's a sound choice.
Reissuing certificates shouldn't be a problem since Let's Encrypt has a rate limit exception for renewals, though if I were managing 65,000 servers I'd be a little hesitant to put that kind of burden on Let's Encrypt's infrastructure without contacting them first, just as a matter of courtesy.
https://community.letsencrypt.org/t/rate-limits-fixing-certs...
it’s less secure — you’re trusting (many) third parties with your security and relying on the security of DNS
it’s less flexible — you can’t sign certificates with internal names (e.g., *.cluster.local) and certs must be for 90 days, etc
it’s kind of hacky — you have to work around rate limits and whatnot because Let’s Encrypt wasn’t designed for this use case
The advantage is it’s easier. But that’s arguable. What the article describes isn’t easy. Using something like cfssl (https://github.com/cloudflare/cfssl) or vault (https://github.com/hashicorp/vault) or step certificates (https://github.com/smallstep/certificates) (which I work on) is probably easier and definitely better for internal services.
For service-to-service stuff and APIs the number of TLS clients that respect pinning is approximately zero.
Either way, the other problems remain, and running an internal PKI is really pretty easy.
Also, using Let's Encrypt doesn't stop you from certificate pinning.
There are far too many ways to self-DoS accidentally with HPKP. Also, an attacker who briefly gains control of your public DNS or web server can DoS a hostname semi-permanently.
You're trusting those third parties regardless of if you use your own PKI, because browsers already trust all those root CAs anyways.
I remember seeing a good tut on this once... but can't find it atm.
Any pointers?
Took me a while to figure out that Lego indeed supports BIND (via the RFC2136 DNS provider).
(Secret keys and private keys aren't the same kind of thing. lyrics to U2's "The Fly" helps me remember, "A Secret is something you tell one other person, so I'm telling you" - somebody else needs to know the _secret_ key for it to work, but nobody at all knows the private key)
A CSR is a signed document which says "I want a certificate for this identity with this public key". The signature proves you know the corresponding private key, but that key isn't transmitted anywhere.
If you're willing to have relatively long lived keys (2-3 years isn't too scary for 2048-bit RSA) you can generate a private key and a CSR once, and have Lego or similar CSR-capable software obtain certificates from Let's Encrypt every couple of months with that CSR.
Just search for: TECHNAME let's encrypt
You should be able to find something. With libraries, you should be able to roll your own. If you're distributing keys to a cluster of servers, you'll need to integrate your own solution, or terminate at the LB and trust your internal connections, or use PKI behind the LB.
... and in my opinion fails at that precise point.
I think that SSL/TLS itself is the correct starting point and not a particular implementation. The article could have been titled "Using TLS for internal servers". The LE implementation could have been one of a few.
Don't forget that you can manage your own Certificate Authority lists and really ought to do so if you actually give a shit about IT security. Abrogating your responsibility to MS, Apple, Mozilla, Google etc is way too easy, dangerous and probably immoral.