Hacker News Reader

Comment by Avamander | Hacker News Reader

The first paragraph already highlights the issue I have with it. I don't want to loosen my firewall, some countries and their IP ranges just need not access my server.

View on HN

I have my certs issued to a VM this is not accessible via HTTP(S) anyway, just DNS. It runs a small DNS server which answers to _acme-challenge.* requests "forwarded" via CNAME. As that is the only part of is visible to the rest it has a very small security surface to worry about. It then pushes out the keys & certs to the places that need them (directly in the case of local machines, less directly for some others).

I set that up originally for a wildcard, as http validation is not supported for them and I didn't fancy mucking about automating updating the main bind instances, but it is convenient for the others too and will be unaffected by these changes.

Not a perfect solution for everyone of course, copies of all the keys are in one place for one thing, and it all in one place could be bad or good for maintenance (single point of failure, but single service to monitor & maintain), but worth considering if you expect problems with http validation.

> some countries and their IP ranges just need not access my server

I'm guessing that they won't be using locations that are commonly blocked in this way anyway. And if they do happen to use one, you may be fine as only two of the three external checks need to be responded to.

I've read about this a few times now, but have been unable to find a good resource on how to set such a VM up. Do you have a link or resource somewhere so that I can at least get started?

Still unexperienced with letsencrypt, but I know enough that I cannot use the standard way.

I think what they're doing here is running a DNS server on a VM that only answers to requests for specific ACME address prefixes. Then at the actual domain, say example.com, they may ask for cool.example.com but it's a CNAME record that directs to something the DNS server is authority over, say xyz.io or something along those lines.

Therefore ACME's DNS request for checking via DNS validation is validated directly by the tiny DNS server due to the CNAME record directing traffic to it.

I've never done this my self neither, but I think it would be along those lines.

> I've read about this a few times now, but have been unable to find a good resource on how to set such a VM up.

You first set up a VM and set up your favourite authoritative DNS software on it: popular choices are ISC's BIND and NLnet's NSD. Either will do. Call it (e.g.) ns-dnsauth.mydomain.com, which is Internet accessible only on udp/53 and tcp/53.

You have to then configure that DNS server to serve the domain (e.g.) dnsauth.mydomain.com.

Next you configure the DNS server software to allow dynamic updates. For ISC BIND, you can set up (crypto) keys and use the nsupdate(1) utility:

* https://www.zytrax.com/books/dns/ch7/xfer.html#allow-update

* https://dan.langille.org/2017/05/31/creating-a-txt-only-nsup...

Point your public/external DNS records to your delegated-auth server by having (say) _acme-challenge.www.mydomain.com be a CNAME to (say) _acme-challenge.www.dnsauth.... LE will follow the CNAME and try to do the verification against the record in dnsauth sub-domain that lives on the ns-dnsauth VM.

Then you have your LE/ACME client(s) run a hook script to publish (and cleanup) the dns-01 TXT challenge records:

* https://dan.langille.org/2017/07/04/acme-sh-getting-free-ssl...

* https://github.com/dehydrated-io/dehydrated/wiki/example-dns...

The LE client goes to the LE API, gets a verification token/nonce, executes the the hook script to push the TXT record to ns-dnsauth, the LE folks verify the record, the LE client (ideally) cleans up the TXT record, receives the cert for the LE API, puts it in the correct path and restarts your (web) service(s).

Someone actually wrote a limited-functionality DNS server that allows for pushing of records via a REST API for this purpose:

* https://github.com/joohoi/acme-dns

This way the 'heavier' BIND/NSD software doesn't have to be used, as those have more features than are needed.

For publicly-accessible infrastructure:

4-6 instances of pdns authoritative for a domain, and pdns recursor running locally for each box. And Cloudflare free tier while revenue can't justify rolling-out Varnish and other locally-deployed capacity/DDoS mitigations.

It may also be a better idea to push DNS updates via configuration management or driven from something like Envoy so there's a history and a single-source-of-truth (SSOT) to point-to rather than multiple people doing manual tinkering, which is a labor-intensive, antiquated approach.

Is "pdns" PowerDNS?

* https://doc.powerdns.com/authoritative/dnsupdate.html

If we're just talking about issuing certs, I don't know why one need 4-6 instances for serving the dnsauth sub-domain.

There are a fair few examples out there for hooking acme-dns in as the DNS server with the standard certbot tool. SO search for acme-dns for more info.

Though that isn't actually how I do it (I'm using dehydrated with my own hook script to update a bind9 instance) but probably would be if I started again from scratch.

You don't even need the DNS to be local and expose anything at all to the internet. You can just as well push the TXT records to your usual DNS through your registrar's API. This generally just works.

Agree. Though I would rather state it "just" works. You need to watch for DNS propagation, particularly if your DNS provider has some sort of CDN-like features (which is the case for instance of OVH) which makes the timing of the propagation non deterministic and non observable (you will likely be served by a different DNS server than the let's encrypt bot, how can you check it has propagated?).

Let's encrypt will only check the DNS entry once, if it doesn't find it, it fails the authentication process and doesn't retry (contrary to the specs).

I still run my own DNS servers, so for me "your registrar's API" is directly poking bind and I only have myself to blame if anything goes wrong! Keeping the little DNS service for the certs means if I break that the rest stays sane. I also don't have to worry about delays transferring changes to zones if there are any intermittent network issues.

And how does filtering on the IP range of a country prevent traffic from that country? Or are you also blocking every known proxy and VPN service? Are you sure that your strict firewall even provides the security you wish it did?

Either way, letsencrypt will probably have a somewhat fixed pool of these challenge validation IP addresses/ranges, so adding those to a white list should probably work with a strict firewall.

I'm pretty sure that even a 50% mitigation is still valuable, and a remarkable number of attacks are completely unsophisticated. Having gone over server logs specifically looking at this sort of thing, I can tell you that a lot of attacks absolutely do appear to originate in countries that are vanishingly unlikely to have legitimate interest in the site (this was a small local business in the US, with logs showing a lot of `/wp-admin`-type attempts from Asia/Russia). So yes, blindly banning unexpected countries is very likely to reduce the number of attacks that you get, and if nothing else reduce log spam. It's a bit like changing your SSH port: easy to bypass, but it still reduces attacks by quite a bit.

>but it still reduces attacks by quite a bit.

But that's a wrong metric to optimize for. Cutting down the number of "attacks" by 90% doesn't improve your security. If your server has a vulnerability that can be picked up by script kiddie scanners, changing the default port only delays the inevitable.

I think the idea isn't to increase security, but to decrease log spam from attacks that are attempted but won't work.

For example, if you don't have WordPress, then all the attempts to access /wp-admin will never work, but will fill your logs with 404 errors.

It does improve security by improving the SNR in logs.

I don't agree with this at all - 'delaying the inevitable' is 90% of security; it's the basic assumption when practicing defense in depth.

You scan for and patch known vulnerabilities, and assume that some unknown ones still remain. You deploy WAF to block some of the unknowns or at least make them harder to exploit. You harden the host and segregate the network to make it harder for the attacker to move laterally when they manage to exploit something anyway. You use SIEM or just regular log review to hopefully catch attackers that have been delayed by your defensive measures.

Putting your service on a non-default port is a perfectly valid measure as part of a larger defensive strategy. It has its pros and cons, you've got to be aware of what threats it mitigates and what it doesn't, but it can be useful.

> I'm pretty sure that even a 50% mitigation is still valuable

It’s about as useful as wearing a condom 50% of the time.

Which is to say, sometimes valuable, sometimes "oh crap!"?

I find that a very good, low cost, low friction measure is to do IP based mitigation only for sensitive ports (ssh, rdp, smb, etc). Ideally you implement an IP white list that you store in a safe and reliable place (cloud storage?). And your servers refresh that IP list every 5 minutes and modify the firewall if it changed. Easy to implement.

Only you can talk to sensitive ports. And the server is available to the rest of the world for non sensitive things. And if you connect from a new IP, within 5 min you have access to the server (I have a scheduled task that updates the IP list with my current IP so I usually don’t even wait).

That kind of sounds like you are implying cutting down 50% of attacks is the same as cutting down 50% of the risks. But not all attacks are equal. Only a very small portion of the risk comes from the 50% you blocked.

If you weren't trying to say that, my mistake.

It blocks unsophisticated resource waste mostly, as other commenters have said, just like changing an SSH port would.

> Either way, letsencrypt will probably have a somewhat fixed pool of these challenge validation IP addresses/ranges, so adding those to a white list should probably work with a strict firewall.

They won't publish the ranges as far as I've read.

Then use a different method of validation, like DNS?

DNS validation once setup is so much easier to work with, for all use-cases I can imagine, I will never go back to HTTP-based authentication.

Really, do give it a try.

Impossible (to do safely) with most providers I've seen, even CloudFlare doesn't offer a properly limited API key.

So delegate the validation with a CNAME and do it somewhere that’s not cloudflare. They provide signing with their own CA for origin certificates so I don’t really know why you’re using Let’s Encrypt at all.

Your idea of just randomly blocking access from certain IP address ranges doesn’t really provide you with any security at all. If you’re worried about rusaian hackers or whatever, most people exploiting anything have access to botnets with whatever bespoke IP address ranges they need to bypass those sort of rules.

In anti fraud we see this commonly, people using stolen details will happily get better matches with GEOIP than the legitimate users of the credentials. Blocking specific countries IP allocations is just providing a false sense security on your part.

> So delegate the validation with a CNAME and do it somewhere that’s not cloudflare.

Like? For what price?

> Blocking specific countries IP allocations is just providing a false sense security on your part.

No, it's a preventative measure. Just like changing SSH to a non-standard port reduces pointless attempts.

> Like? For what price?

If you own example.com, you can delegate to dnsauth.example.com for $0 (or simply the price of a Internet-facing machine that has DNS open).

Say you want a cert for www.example.com. LE will check for ownership by looking up _acme-challenge.www.example.com. Instead of having a TXT record with the nonce, _acme-challenge.www is actually a CNAME pointing to _acme-challenge.www.dnsauth--where the TXT nonce lives.

The DNS daemon that is authoritative for dnsauth can be the traditional BIND, or other software:

* https://github.com/joohoi/acme-dns

This is often called 'DNS alias' mode:

* https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo...

* https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...

I did not ask "how", I wished to know who supports a DNS service like that and for what price.

The parent stated that you can run your own DNS server temporarily for the cost of the hardware to run the server and shut the DNS server off after the certificate has been issued. The cost is basically free.

Any DNS service which allows you to create a CNAME RR supports it. You delegate the subdomain to any DNS server you wish.

This isn't some special "Let's Encrypt DNS forwarding mode" that DNS providers have to explicitly support. It's simply part of "how DNS works".

> Any DNS service which allows you to create a CNAME RR supports it.

And which of those also have an API that is supported by Certbot?

I would really like names where a setup like this has been tested and works.

> And which of those also have an API that is supported by Certbot?

Certbot allows for hook scripts, and you can use a utility that can talk multiple APIs:

* https://github.com/AnalogJ/lexicon

> I would really like names where a setup like this has been tested and works.

The guy who runs BSDCan and PgCon uses it for his personal stuff as well as FreshPorts.org, etc:

* https://dan.langille.org/2017/05/31/creating-a-txt-only-nsup...

* https://dan.langille.org/2019/02/01/acme-domain-alias-mode/

He used acme.sh, though I'm more partial to dehydrated:

* https://github.com/dehydrated-io/dehydrated/wiki/example-dns...

We use it at work, but I don't want to dox myself. :)

> I did not ask "how", I wished to know who supports a DNS service like that and for what price.

And as I stated in the very first sentence, it is self-serve:

> If you own example.com, you can delegate to dnsauth.example.com for $0 (or simply the price of a Internet-facing machine that has DNS open).

We do this at work: our main registrar does not have a restricted API, so we have a sub-domain that lives on a DNS server in our DMZ. Internal ACME clients update the desired TXT records when asking LE for a cert.

The cost is the price for keeping a VM running and updated, which for us is minimal since it is on our private cloud.

Even with CloudFlare in front of your servers, it is still valuable to use Let's Encrypt certificates. You can turn on Full Strict SSL validation to the backend and reduce another attack vector. It's an unlikely attack vector to be exploited but it's also a trivial amount of work to implement.

Every layer of security makes attacks that much more costly.

Like I said cloudflare signs certificates for your origin as well.

And if you always use Cloudflare (or at least are sure you'd have days-weeks not minutes-hours between making a decision to cease using Cloudflare and actually executing) then it's actually safer to tell them you'll use their Origin certs rather than a public CA as well as likely being easier.

Why can't you do this safely with AWS? You can restrict the API key to write only to the zone _acme-challenge.<your domain>.

Smart way to do it, but it will set you back $0.5/month. You can likely do it cheaper with Lambda and API Gateway, but you'll have to invent the secret sauce yourself.

I'm always a little surprised when there are short comings in IAM like that with route53 and records. It seems like a natural thing to be able to control, but for some reason you don't have resource level controls on hosted zones. It's all or nothing.

This is now supported as of a few weeks ago! I just set it up for a new domain using a cancel-able API key.

I've just checked it, but with API tokens, I can only allow 'edit' rights on DNS records for a specific domain.

There is no way to create a token allowing access only to _acme-challenge record.

You can use the CNAME trick to canonicalize all ACME challenge requests into a subdomain you reserved for this purpose and then give the tokens access to that subdomain.

Let's Encrypt is obeying normal DNS mechanics, so when they ask for a TXT record for _acme-challenge.cat-photos.example.com and get a CNAME as a response, they'll ask for the TXT record for the name in the CNAME answer instead. If that's cat-photos.cert-issuer.example.com then a token valid only for the sub-domain cert-issuer.example.com can write that TXT record.

You sort out the CNAME once, probably when creating cat-photos.example.com or setting it up to get a certificate, and then afterwards the API token is enough for automation.

If you have that kind of security profile that you need to lock down what ip addresses that can make a port 80 connection, and can't use the api of the dns provider because of security considerations, then how is giving a third party dns provider full control of DNS acceptable within the threat model?

Run your own DNS server and get a registrar lock. If that is not feasible, and consultants are too expensive, I would look into the more expensive dns providers that provide custom interface that fit the threat model of your system. If that is also too expensive then I would take a second look at the risk analysis and recalculate the cost of each risk.

You can use pre and post renewal hooks in order to temporarily disable those firewall rules.

You could just loosen the firewall for a few minutes before and after you requested DNS validation.

Yeah, looks like you have created a problem there for yourself. Back to other SSL providers for you, then. Unless you want to jump through some hoops.

I’m with you I have China completely blocked from my servers. There is no reason for anyone in that country to access my apis

Recently, the U.S. government accused four Chinese nationals in the Equifax hack. The U.S. says they made us of something like 34 servers in 20 countries to carry out the attack.

Do you really think you're stopping anyone in .cn who wants to actually connect to your server from doing so?

Proxies and botnets are a thing.

I don't care - my country block is a two second thing that cuts off the vast majority of regular people hitting my server.

"Limiting attack surface" is also a thing. Claiming there's no benefit is empirically false.

Reply on news.ycombinator.com