Hacker News Reader

Comment by octorian | Hacker News Reader

I really wish Let's Encrypt had a solution for those of us with lots of "internal" devices that neither need public access nor are capable of running an automated renewal job (routers, switches, IoT gizmos, etc).

All of these things come with half-baked self-signed certs, and web browsers and mobile devices these days are making it increasingly difficult to actually stomach such certs for anything at all.

I wish the "SSL All The Things!" crusaders would consider this use case once in a while, and actually offer a usable solution.

(Some of us could go through the trouble of setting up a private CA to deal with this issue, but getting everything to trust your own personal CA root seems like almost as much of a hassle at times.)

View on HN

I could not agree more. Information Technology, for reasons unknown to me, seems to ferment a kind of extremism with a large helping of "if you are NOT doing it this way you are what is killing the world," and the "SSL All the Things!" people are a great example of it. I've heard people suggest that you should just dump your web-serving software in favor of a different webserver that works better with Let's Encrypt, nevermind that a lot of people in the real world are running niche proprietary software that has very specific webservers they are tightly-integrated with. And -- it has been argued to me -- the solution was to not just get in that service sector in the first place.

Most of the time, you can stick Nginx or HAProxy in front of it all, and terminate TLS there, with the last part on your interal network being HTTP.

Sounds great until you consider threats inside your network

I think many here don't get that not working in a tech startup is very different.

The IT budget is shrinking every year instead of increasing with company profits and sometimes a 3 person team needs to handle 300 users plus those machines that nowadays are tightly interconnected. These 300 users are essentially a threat to the company because they often click on anything they get per mail.

So you need to encrypt everything as much as possible to decrease the attack surface inside the network.

> These 300 users are essentially a threat to the company because they often click on anything they get per mail.

I agree, but I have to mention as well that in general all teams (not only IT) got reductions of budgets and amount of employees and in the end everybody is very stressed most of the time, which in turn makes everybody pay less attention to what they're doing.

In my case (IT) last year I received an email with the title "ACTION REQUIRED" and I just blindly opened it and clicked on the link of the document which I was supposed to fill out, and (luckily) it was just a well-crafted internal test related to phishing (which then of course I failed => I then had to take an online course) - in my opinion the problem is that nowadays we're permanently under pressure to deliver/react/do stuff with very short lead times and it's therefore hard after a while not to start to unwillingly forget best practices related to security.

This year the company I'm working for has started as well to use quite often external companies (websites) to provide some services => doing that makes identifying phishing emails even more difficult for me.

You can terminate SSL at nginx and forward over socket or local port to service bound only on localhost. Then you can LB over the nginx's. You can even deploy the same certs to all of them.

Assuming the device isn't some embedded system or router.

It is also assuming that Baldrick, Hacker Extraordinaire, hasn't installed his brainchip in all your developers' brains. That is true.

You can use a secure link between the front-end nginx and the backends (VPNs, another TLS connection with a cert signed by an internal CA, IPsec, ...)

If it's running Linux and you can add HAproxy or Nginx directly to it you can sidestep that issue as well.

As gp said: "different webserver that works better with Let's Encrypt"

nginx is one of the most common examples of that

> lots of "internal" devices that neither need public access nor are capable of running an automated renewal job (routers, switches, IoT gizmos, etc).

How is that a CA's reponsibility? I wish vendors would setup an appropriate API for dealing with that on their devices. Quite often you need to enter strange web interfaces in order to setup a certificate.

Let's Encrypt needs to verify you're in control of the hostname. This can happen with either http or dns verification. DNS verification with certbot requires you to setup almost nothing (i.e. you don't need the machine running certbot to be exposed to the public internet) if you're using a supported DNS provider (there're many) for the zone.

So, the missing part is NOT in let's encrypt infrastructure. It's in the device API. You can launch the certificate renewal script just about anywhere (e.g. a small server or vm within your network that can access the internet); but then, how would you automatically deliver the certificate to the device? That's the missing link.

> How is that a CA's reponsibility?

The request wasn't (only) asking for a solution by the CAs but by the '"SSL All The Things!" crusaders' which includes creators of browsers and mobile devices enforcing stricter rules and a request to take these things into more consideration.

In my words: We need solutions for non-technical individuals to run a somewhat secure environment locally without being confronted with security warnings everywhere. A CA can play a role in that, but probably not fully solve it.

Are you joking? The primary reason why one would want one certificate per device is because you are directly connecting to the device instead of letting the device connect to the vendor API which is already behind HTTPS. Your idea will just lead to a lot of bricked IoT devices once the vendor takes his certificate renewal API down.

The device can present an API as well.

I don't understand. The device should expose an API (instead of a webinterface) to generate csrs and upload certs (ideally the key should never leave the device). What is the vendor api?

One of the challenges in this space is similar to captive portal detection. It’s not enough to have a solution, you need the vendors of those devices to adopt the solution.

Unfortunately, just like captive portals want to avoid detection by your OS [*], the vendors of these devices want to do the least possible work, for the lowest possible cost, never updating or supporting the device once sold. Any new solution is more work for them, especially early on when things are still in development/in flux and may require changes over time, since the problems in this space are actually rather difficult.

It’s not that this problem isn’t important, but rather that there’s no clear good path here. However, even still, the alternative (of HTTP) is much worse for the Web, so TLS is still promoted aggressively.

[1] The reason captive portals want to avoid OS detection is because the OS-spawned UI generally doesn’t have access to your browser cookies. The ecosystem has mostly ossified around a few captive portal vendors, who want your cookies either for advertising/demographics/traffic, or want to offer “Sign in with Social Network” (to both track and... advertising). So they want you to use your main browser, not the OS UI, and thus want to actively avoid OS detection. It’s sad.

The very long term good news on captive portals is that users hate them, so they're doomed on that basis.

Doesn't matter. Users aren't the ones the ones deciding to use them or not.

> One of the challenges in this space is similar to captive portal detection

I really wish someone had thought of that problem when defining the 802.11 specs. Though I suspect the idea hadn't even occurred to them at the time.

This was much worse back in the mid-00's when WiFi was a newer use case. Seemed like flipping open my laptop in a public place back then would suddenly cause all the network-connected widgets on my desktop to go absolutely berserk as a result of portal nonsense.

Well Let's Encrypt has a solution, it is DNS authentication, which doesn't require Let's Encrypt to establish a direct connection to the device. But you still need a way to automatically renew the certificate (your DNS must have a supported API) and deploy it automatically. But that's not a Let's Encrypt problem to solve.

I use this solution myself.

I have a domain name I own and only have A records configured for use on internal DNS.

Externally it is managed by cloudflare, and certbot uses the cloudflare plugin to renew certificates before deploying certs to my hardware over telnet

How I read this comment, presented as an audio-playback-style progress bar:

  [--------------------------------------------??]
     ^ Interesting!   ^ Huh, that's a good i-- ^ Wait, what?

Deploying via telnet is reality for a lot of kit

> But that's not a Let's Encrypt problem to solve.

But it's a problem they permitted to arise by enforcing 90-day expiry

An internal web config interface could happily use a 10-year cert

Browsers are also enforcing this, so again not really a let’s encrypt problem.

But create your own root certs. That’s a much better route if you want to never update your certs on the device.

Do browsers allow > 13 month validity for certs from private CAs?

Either way I’d rather not trust my own CA as I’m not capable of keeping such a powerful tool secure.

For private CAs you can do whatever you want. It is only for public CAs that browser enforce short certificate expiries.

Even when you have a private CA, you still have the issue of getting everything in your environment to trust it. Even in an enterprise setting, from my experience. Most workstations and servers are managed and have the root cert in the trust store, but then you have a long tail of appliances and containerised applications, or OSS software that doesn't use the OS trust store or BYO devices.

With publicly trusted certs you instead trade the trusting-CA-issue for the issuing-issue.

I'm starting to think that it's easier to use publicly trusted certs with FQDN for internal use. Some kind of ACME-proxy, and portal, which is publicly accessible and can respond to ACME inquires. Maybe in combination with some non-LE CA that can issue long lived certs for certain uses where updates can't be automated (e.g network appliances). But then there's the Certificate Transparency Issue, leaking internal names etc.

The dream would be a LE issued sub-CA limited to *.internal.example.com, so you can have an "internal" CA that is publicly trusted.

That would mean I can issue a valid, trusted worldwide, certificate what will work on your internal site.

Using public domains for internal sites are the way to go in my opinion. As long as you pay the bills you have guarantee that you control your domain. You can issue any kind of certificates and so on. If you have a nice naming scheme you won't leak that much info in then Internet. A lot of companies I know does that, by the recommendation of security teams (due to issues with using .local or .example.com domains).

> If you have a nice naming scheme you won't leak that much info in then Internet.

IME even if you leak the name of every internal service, you're still more secure than training all users to ignore certificate errors, which is what internal CAs do 85% of the time.

Maybe I was unclear, but I was wishing for a situation where the Name Constraint extension was wildly enough used that LE would issue scoped sub-CA certs for *.whatever.yourdomain.com, which I hold the private keys to.

But yes, I agree with your larger point. However, "nice" naming scheme might conflict with "secure" naming scheme. E.g, perhaps admin-{not-yet-released-product-line}.acme.com is the most resonable domain name, but it would be unsuitable if it was publicly announced via CT logs.

Any way, having the domains in the CT logs, and naming them considering that is probably a tradeoff worth making.

I use DNS auth for internal certificates. I have multiple domains that only have entries on 10.0.0.0/8, mostly as *.identifier.domain.tld. With DNS auth they all have valid certificates from LetsEncrypt.

Do you actually have that or are you suggesting it a solution? Unless something changed you can't get certs for subdomains unless they they are the public suffix list and getting on that list is non-trivial.

Maybe I'm misunderstanding something, but I've been doing what I suggested for years.

I just verified, the domain that I use doesn't have a single A entry that isn't on 10.0.0.0/8.

I actually only have 6 A entries, that are the in the form of

    a.domain.tld  
    *.a.domain.tld  
    b.domain.tld  
    *.b.domain.tld  
    c.domain.tld  
    *.c.domain.tld

Then I use Caddy server as a reverse proxy on the internal network, which is configured to do DNS challenges to get certificates. Here's the plugin for AWS Route53 for example: https://github.com/caddy-dns/route53 - The challenge just verifies that I have control over the domain through DNS and provides a certificate to me no problem.

It's been working perfectly for few years for us on our internal networks. Was the OP asking for something different? I'm not entirely sure what the 'public suffix list' is for subdomains, but I definitely have a valid certificate right now for *.a.domain.tld, served internally and provided by LetsEncrypt.

https://letsencrypt.org/docs/rate-limits/

The public suffix list is (or was) used to tell if foo.domain.com is considered different user than bar.domain.com

If domain.com is on the list then foo.domain.com is different than bar.domain.com. If not then it's considered a single user and the rate limits apply.

Those rate limits are fine for almost all internal IT uses. The 50 certificates per week limit does not count renewals, that's 50 new hostnames added every week. Maybe larger shops need to roll out certificates a bit slower in order to not exceed that limit but it's still a pretty generous limit.

The public suffix list only affects limits with letsencrypt. The limit is still quite high IIRC, you just need to have proper backoff an smoothing between devices.

DNS Auth doesn't work with split-horizon DNS though, since the LE process will update the internal view instead of the external.

It sounds like you've put all your RRs in one flat zone visible internally ad externally but then that breaks device network portability, since mail.example.net on 10.x.x.x won't be reachable once outside the internal network

ACME and LE were really developed without an understanding of how certs work in the IT World. They were developed by folks who spin up a service on AWS and think that's how the world works.

SSL externally points to 123.45.67.8

Your internal DNS responds on 10.20.30.40

SSL very works fine as it validates Mail.mycorp.com, not the IP

This fails as DoH becomes more common and your dhcp server saying “use dns 10.1.1.1” is ignored, but then connections to your mail server fail too.

> (Some of us could go through the trouble of setting up a private CA to deal with this issue, but getting everything to trust your own personal CA root seems like almost as much of a hassle at times.)

This is the actual problem... setting up your own CA for internal networks could be automated relatively easily.

The problem is not the self-signed cert or CA. The problem is managing trust on the devices themselves.

Imagine you want to trust only _your_ self-signed cert or CA root for a specific service. Good luck making that work.

This issue should be so common that there should baked-in functionality in every piece of software to allow for this. It's often totally missing or implemented incorrectly (self-signed certs or custom CAs are often trusted in _addition_ to the system CA roots!).

I use mkcert [1] with great success - it's easy to setup locally trusted certs which I load onto my router and other devices.

[1]: https://github.com/FiloSottile/mkcert

That seems like a nifty tool, but it doesn't seem to do anything to address the difficulty of getting your private CA trusted on more than one of your devices. I'd like to be able to access my router and switch configuration interfaces from more than one client PC, and perhaps even from a smartphone.

Same here. I run OpenWRT on my home routers, and your comment prompted me to do a quick web search. uacme looks promising for luci. I’d love to hear HN experiences with it.

You can copy the (public part) of the generated root certificate over to your other computers and smartphones.

Yes, that is the hard part.

FWIW, tapping on .crt files in Chrome offers to open them with the certificate manager on Android, where it's then straightforward to install them. Unfortunately this also produces some very irritating (non-removable) notifications... >.<

...And I can't deny I was fetching the certificate from my PC across the room over HTTP, not HTTPS.

Getting your private CA trusted on a smartphone seems to keep getting more and more difficult and/or annoying with every iteration of the various mobile OSes.

If it was possible to make a private CA cert that was only valid for hosts under a single domain, and then for mobile devices to STFU about you wanting to install it (as it wouldn't be as "risky" from their POV), life would be a little easier.

It's actually possible to do so, but many non-browser/mainstream cert validation libraries have a habit of just ignoring that restriction.

https://github.com/acmesh-official/acme.sh this works for me for most of my devices at home. pfSense and Proxmox both have plugins for it. Each device requests its own certificate for a host on a private subdomain of a domain I own e.g. hostname.private.domain.com

That still only works if your device is able to reach the internet. We have a bunch of servers which aren't directly on the internet, because they don't need to.

You can run acme.sh on another device using dns-01 authentication. Then you can move the certificates to your servers in some way. This would mean one machine having access to the internet and the private servers though.

But also, LE certs expire after 90 days, meaning this manual process needs to be repeated every <90 days.

Though, I'd be curious to know what the threat model for such a service is. Why is it not allowed to connect to the Internet? Can a different internal machine connect to it? And if so, is that machine allowed to connect to the Internet. What happens if someone connected to that machine and opens a tunnel through their machine to the Internet (if at all possible). Understanding the threat model will give you a solution on how best to use dns-01 challenge to generate certs and keep them updated.

Alternately, you could use a fast CA solution like Netflix' lemur or step-ca or even just openssl, make your own CA, and distribute it to everything. If your threat model is such that that stuff should never connect to the outside world, why risk it by repeatedly moving things in from outside? Just generate a closed CA for your closed environment, trust it once, and move on.

Reminds me how either Xiaomi, or some other fairly big smart things co. bricked few millions of their smart lightbulbs when their root cert expired, and they had no valid cert left to sign a new firmware.

Maybe some kind of wildcard approach would be helpful, where you generate your Let's Encrypt certificate with a wildcard, and are able to distribute it automatically to every device via a standardized API.

That way only one device (possibly a Raspberry Pi) would actually be confronted with the task of generating the certificate and validating the verification requests from LE's servers.

This is what I'm doing at home: on a public VPS I have a custom made DNS/HTTP-server, which is pointed to by the NS-Record for the subdomain int.example.com by my DNS-Provider for my domain example.com. So everything but int.example.com is under control of the DNS-Provider, int.example.com is under the control of my custom DNS/HTTP-Server.

The local Raspberry Pi invokes certbot, gets the authentication tokens from LE's servers, POSTs them to the DNS/HTTP-server via HTTP, LE's servers query the DNS-server for the TXT-Record and certbot then continues with its work of storing the new certificates. Finally a script then distributes these certificates to the local HTTP-servers.

The thing about these certificates is that they contain a wildcard certificate for * .int.example.com, and this way I get a wildcard certificate covering server-1.int.example.com, server-2.int.example.com, ...

If the IoT-devices would have a standardized API to deploy the certificates to them, then I could access the ip camera via https://ipcam-1.int.example.com.

The public DNS/HTTP-server does not respond to requests of any of the * .int.example.com subdomains. The local Pi-Hole has all the DNS-records for those subdomains.

Let's Encrypt revolves around short lived certificates and automated solutions for updating. I think your best bet is having an internal "service" (could be in a docker container), that do the updating. If you can SSH into the device, you can write a script that does it.

look into Cloudflare’s cfssl. it’s really great if you just want a simple internal CA.

I point a bunch of subdomains to my local IPs using https://nip.io and authenticate them all using a wildcard Let’s Encrypt certificate. Works great.

A very detailed tutorial with screenshots for those of us with no clue would be much appreciated.

You can use dns for certs instead of web/http verification.

> Some of us could go through the trouble of setting up a private CA to deal with this issue

I've read about doing that for development purposes. But it sounds complicated. Therefore I wonder couldn't there be an open source solution that produces an npm module which when I run it creates a private CA for me?

I mean if it is complicated to create a private CA, why not automate that task?

"mkcert" is a tool to automate this: https://github.com/FiloSottile/mkcert

I've deployed step-ca internally and have my root certificate added through cloud-init for new servers. Also supports ACME https://smallstep.com/docs/getting-started/

The solution is somewhat simple - have Let's Encrypt (or any root CA) issue intermediary CA limited to your domain. You "only" need devices to trust the root CA and could issue certs at will. The problem is that this use case was/is a threat to CA business model and is not really supported in the current cert infrastructure.

Isn‘t that mostly because the necessary X.509 extension isn‘t widely supported?

Of course, this seems like a chicken and egg problem, but my point is that adoption would take a couple of years in practice.

Yep, as long as relevant specs are not implemented by the majority of things touching trust (including appliances with very long update cycles), this is, sadly, "no bueno".

Reply on news.ycombinator.com