Hacker News Reader

Comment by dullgiulio | Hacker News Reader

If you start an nginx reverse proxy and it cannot reach upstream, nginx won't start at all. Only way to fix this is to pay.

Some friends got hit by this: they forgot some upstream from a domain that got removed, at the next nginx restart the whole server didn't come up.

Basically you have to pay for basic error handling. It's not "more advanced deployments" at all.

View on HN

I have been using nginx pretty heavily in a wide variety of use cases for coming up on 10 years and I have never encountered this issue. I have also always used the free version of nginx.

I of course don't dispute that this limitation exists, but I'm just pointing out that the need for this type of error handling is, in my experience, quite rare.

There may be some very specific use cases where this is a deal breaker, but I would say the claim that the free version of nginx does not include "basic error handling" is a pretty grotesque exaggeration.

The scenario described only occurs if they’re using host names for their upstreams, and that hostname is no longer resolvable. It results in an error during config parsing, preventing reloads and fresh starts. Moral of the story: don’t use DNS names in upstreams unless you’re sure they’ll always resolve I guess.

That's tough: we do docker compose, so upstream = named container, and so init can be wonky (ex: dev). We have started strangling Nginx out for caddy bc this + LE (no pay gate on ssl autorenewal). These are increasingly basic things for modern apps.

>> If you start an nginx reverse proxy and it cannot reach upstream, nginx won't start at all. Only way to fix this is to pay.

I don't understand - if my upstream is down my nginx returns the usual "503 Service Unavailable". Are there any particulars to the configuration you are describing?

The problem isn't when the host is down, it is when nginx can't lookup the address for the host. If your DNS records are OK and your resolvers are working this isn't a problem, but if nginx (re)starts while you are having connectivity issues or if a DNS record that your config relies upon has been dropped it won't start any service until that one is edited.

Workarounds include using addresses not names (not practical at scale), running a local resolver that will hand out the last known address if it can't see the resolvers it normally forwards requests to or gets an NXDOMAIN response (extra hassle and technically breaks DNS so make sure other things don't use that resolver), or just ignoring the problem because it very rarely happens (though for some setups it is more likely than in others).

In my (current) uses of nginx it isn't a problem at all, though a concern springs to mind that might affect me later: if it only fails on startup, does that imply it isn't ever refreshing name->address mappings at all in normal operation so that if a server it is proxying for moves address that proxy config will start to fail until nginx is restarted? Any nginx experts want to comment so I can be lazy and not test or otherwise research this myself?

Huh. I don't think I've ever encountered this because I've only used nginx to server content or applications. I've always had it behind haproxy, which does have an option to allow haproxy to startup even if a backend host name doesn't resolve.

You describe an environment that is quite scaled up. I would assume that if there is scale, there is money. And a license to the commercial version is what, 2 salaried hours a year? It is easy to save time and money. I am sorry, but I cannot see this as a big problem.

We'd strayed off topic and were discussing an issue in nginx. Basic per-instance subscriptions for that are $2,500/yr each for small numbers of instances which is somewhat more than a couple of salaried hours.

Still not a large amount for a company with a profit making product, of course, but companies with profit making products are fewer and farther between than companies would like, and for a one-man-band or other small business trying to boot-strap something bigger, there isn't the same drop/ocean ratio.

Also, the implied problem (this still needs confirming/contradicting, so I might be barking up the wrong end of the stick) that it may not be updating name->address resolution regularly during normal operation (otherwise why does this only cause failure on start-up?) might not by fixed by the extra modules enabled by the paid licence.

It only happens when nginx is starting and the hostname of the upstream doesn't resolve, doesn't have to be reachable.

I believe just specifying a resolver should fix it, but another common workaround is to use a variable for the hostname

> Only way to fix this is to pay.

Nope.

  resolver 1.1.1.1;
  set $resolved_upstream whatever-it-is.lan;
  proxy_pass http://$resolved_upstream;

I am for sale, FWIW. ;) The first one's on me.

This is particularly annoying when using nginx to sit in front of a bunch of uwsgi (python) docker containers. if one of the containers doesn't start for some reason, nginx will abort.

Presumably you could write some basic error handling and distribute under the same or even stricter oss terms? It would be fairly simple to 'uncripple' it in other words, either with contributed code, or to wrap the upstream in haproxy itself to ensure availability.

You could but this is not in question here.

>> Their paid features are for larger and more advanced deployments.

This is simple not true.

I avoid this by running configtest before the reload - it checks it can get the upstream ips.

Reply on news.ycombinator.com