Avoiding the top Nginx configuration mistakes
nginx.com
nginx.com
I understand the rational for not changing old defaults in fear of breaking existing configs. However, maybe a release or two of deprecation or better optimized defaults under new names or a new (kinda gross but...) strict mode which is opt-in but is the new default for new instalations.
1. Warn the user on startup of potential misconfiguration 2. Allow error_log off to work, as users are expecting it to 3. Enable keep alive by default 4. Acknowledge the confusing syntax and find a better solution than blaming users 5. Warn the user on startup of impact of proxy_buffering off 6. See 4 7. See 1 8. See 1 9. Fix the proxy hash algorithm so that it hashes the whole address (why only first 3 octets?) or give it a new name; see also 1 10. Detect possible optimizations on startup and implement automatically; why ask the user to config something that can potentially be detected automatically?
I get that NGINX is powerful. But I hate dealing with it. Every. Single. Time.
That's not to say it can't improve...
Why? Our API is idempotent, but we needed to use POST instead of GET to let the browser send a request body. Our API is very slow, so we put NGINX in front of our API to cache responses. We used something like proxy_cache_key "$request_uri|$request_body"; I don't think I was completely remiss in thinking that $request_body means "request body". Testing showed that it worked just fine. In production, some users made requests that were larger than our tests - and then $request_body is empty, and users will get each others' responses. This behavior is helpfully documented as follows:
"The variable’s value is made available in locations processed by the proxy_pass, fastcgi_pass, uwsgi_pass, and scgi_pass directives when the request body was read to a memory buffer."
Stackoverflow is more helpful: https://stackoverflow.com/questions/18795701/nginx-proxy-cac...
profile base2022
It also helps on slowly deprecating the older behavior, you just say "nginx2004 is now deprecated", and people understands what you mean.
And if backward compatibility is important, take a leaf out of Rust’s book and make a nginx-config-version 2; directive or something.
With this set, you can safely use the new defaults. Use it across all example configurations on the website and in distribution packages.
Something like this or a \version "2.12.0" command would be massively useful. The alternative is a sea of undisclosed footguns that are blindingly obvious once you know they exist but otherwise are well-camouflaged. Ok, the other alternative is for every future developer to brainlessly spam a long incantation of sensible values that they swear they'll learn about later just so a Fortune 500 early-adopter company can have smoother upgrades.
While working on that issue, I also needed to set the proxy buffer size. Quick, what's the difference between proxy_buffer_size [1] and the size argument of proxy_buffers [2]? I still don't know, because the docs for each directive sound to me like they're restating the same exact thing.
[0] https://nginx.org/en/docs/http/ngx_http_proxy_module.html#pr... [1] https://nginx.org/en/docs/http/ngx_http_proxy_module.html#pr... [2] https://nginx.org/en/docs/http/ngx_http_proxy_module.html#pr...
The article says "Proxy buffering means that NGINX stores the response from a server in internal buffers as it comes in, and doesn’t start sending data to the client until the entire response is buffered."
But those parameters say nothing about waiting to send data to the client, and in fact some of them like proxy_busy_buffers_size imply the exact opposite.
The parameters mentioned look like they're about tuning the relative memory costs of sending down the first bit of data in a response, which may contain enough information on the client side to do something interesting. proxy_buffer_size looks like it's just the first buffer, while the size argument to proxy_buffers applies to all the buffers.
I don't know why it happens to me so often, but I will set up nginx containers that proxy to some load balanced service (via DNS), and then one day the IPs will change and that service would break because I forgot to set resolver.
'resolver' still does something in the open source version, but requires some configuration contortions.
More details here: https://github.com/DmitryFillo/nginx-proxy-pitfalls
This post actually means a lot because Caddy became so popular that (1) I assumed whatever I had written had been completely scrapped and forgotten and (2) no one would ever believe me that I made a big contribution "before Caddy was popular". It's one of the projects I'm so happy to see has come so far.
For smaller projects, I've really enjoyed Caddy's "best practice by default" approach to webserver/loadbalancer config where you can generally only choose to remove best practices, not try to build them up from scratch. Has SSL and LetsEncrypt enabled by default, etc. This avoids a lot of the pitfalls of incorrect webserver config altogether, keeps the configuration file really small and easy to read. I wish this style of config was more common.
Sure, LOC is not a great indicator of size, but here's a simple comparison: https://pastebin.com/kmXMX6Yq
And this doesn't include the pain of manually refreshing SSL certificates, certbot always managed to break on me and I could never get auto-SSL to work.
How does caddy compare, performance-wise, to apache and nginx?
Serving http is usually completely I/O bound. The more efficient parser in nginx never mattered in the real world. It was used for the much more readable configuration and the fewer bundled foot guns.
What matters for performance is mostly your tcp settings, your buffer sizes, and that disk logging. And that's only in the few cases where the upstream application isn't the bottleneck.
The common wisdom when I last looked into this was that apache uses more CPU, especially under load. Naturally, I want to use the httpd that's most efficient on CPU since it's a finite resource, I'm running several different applications on that box, and you never know when one of your side projects is going to meet the HN hug of death.
It can be hard to separate common wisdom from nonsense on the Internet. It's easy to test yourself. There are plenty of easy to use http benchmarks out there, including wrk and ab. Just be wary that the benchmark tool itself will use more cpu than the http server. Which on the other hand might tell you everything you need to know.
If you want to spend your cpu cycles more wisely, you should tune your software and network stack to your request and response size, carefully consider caching and pipelining, not mindlessly switch software. For certain applications this will even make sense.
Of course this has zero relevance if your application spits out a megabyte of content with every request.
Not since Apache 2.4 and the latest APR libraries that Apache depends on. Some Linux distributions kept the 2.2 branch for a while and that may be one of the reasons people still associate Apache with slowness.
Also, this is a bit of a personal preference, but v1 Caddy felt like maybe a bit easier to get started with than v2 Caddy, though sadly was abandoned. I'm sure that there were good reasons for doing it, but the few forks that started out from it with the intent to maintain it never really went everywhere, so v2 is the only possibility nowadays, unless you want to maintain it yourself or like dead software: https://github.com/WedgeServer/wedge
Of course, that's not a criticism of Caddy itself, just how we as an industry sometimes need rewrites and that makes us keep up with the churn.
Also, any time someone brings up the malicious fork "wedge", it saddens me. That was done at literally the most stressful moment in the project's history, and it only made things worse for everyone. I really hope people learn to chill out and give maintainers a bit of time to breathe and respond instead of taking such hostile measures like that. (You can probably pretty easily dig up the Github issue with the discussion if you really care to see what went down. Sigh.)
You know, as someone who used Caddy v1 pretty extensively (still probably have it running on some of my homelab boxes), i never really ran into those supposed problems. Maybe they'd manifest in more complex configurations, but as a reverse proxy, file host, web server that's integrated with PHP or even something to allow setting up basicauth, i never found it to be lacking.
That's not to say that Caddy v2 is bad, just that someone for whom v1 worked perfectly might find it a bit cumbersome to move to the newer version, as the old one is no longer supported. Of course, you can say the same about JDK 8 vs JDK 11+ etc.
> Also, any time someone brings up the malicious fork "wedge", it saddens me.
If i recall correctly, it just took the project and rebranded it, which isn't necessarily malicious (for example, aggregating and selling users' data would). That's the nature of open source - anyone who wants to do that, can.
Of course, i think that the fork is also irrelevant because they couldn't actually be bothered to maintain the fork and nobody cared for it, much like how other projects, like Materialize.css, ended up.
For example, here's the original: https://github.com/Dogfalo/materialize
Then someone got upset that the project was abandoned (even though they were taking folks Patreon money): https://github.com/Dogfalo/materialize/issues/6615
They created a fork of their own: https://github.com/materializecss/materialize
Which promptly died down because there just wasn't enough interest in maintaining it. That's just how things go sometime.
That's not user error. That's lazy programming and a bug. If your code segfaults because of user input you weren't expecting, that's on you as a programmer.
Why are the empty if blocks breaking things outside of them?
Let's Encrypt automates cert renewal, and SWAG automates having to remember how in the heck to set up LE.
If any reader is thinking of writing a tutorial, do check if it's been done before. Likely a grossly simplified example of how to do X while the person reading it has read upteen tutorials on how to do X, Y and Z but still doesn't know where to keep configuration files, or how anything interacts with anything else.
Instead, think of writing Series Of Tutorials That Together Build Moderately Fully Fledged Thing Including Satellite Stuff Like, in this case, Nginx, or SMS, or why XX characters for an email address string, or whatever of the plethora of things to Make It Work are needed.
Some books do this. Not a huge number though, certainly not compared to hit-and-run tutorials.
Call them books. Call them Tutorial Super Series. I don't think the little more time invested would not generate outsized returns.
Does this mean I should just stick with a more managed solution? Or is this a common feeling that I should just learn to live with? For my fun projects I use Caddy with mostly default settings.
Now that said nginx website has a ton of great examples. You can usually copy/paste them into whatever you need pretty easily, and once it's done it's not likely to change. I would especially do this if nginx is your internet-facing load balancer and has multiple upstreams behind it. Plenty of good examples out there.
It looks for the following misconfigurations[0]:
- [ssrf] Server Side Request Forgery
- [http_splitting] HTTP Splitting
- [origins] Problems with referrer/origin validation
- [add_header_redefinition] Redefining of response headers by "add_header" directive
- [host_spoofing] Request's Host header forgery
- [valid_referers] none in valid_referers
- [add_header_multiline] Multiline response headers
- [alias_traversal] Path traversal via misconfigured alias
Now we are seeing the opposite backlash.
As someone who doesnt know much about this topic, I also wish there was an easy answer :/
With software packages such as nginx (in general, not in the context of Linux distributions mentioned above), it's sometimes easier / clearer to start with a completely empty configuration file, and add just enough to make it work (TDD-style), and / or stop it complaining. If you can explain / defend / justify the presence of every line, you stand a much better chance of knowing how it will behave.
I've since moved to Caddy, and uh, wow, this is what it's supposed to be like!
In earlier days a lot of people was under the impression that nginx was much faster for PHP, but that was not true. Apache with mod_php was faster, but the issue was memory consumption for non php files as the php module was loaded on every request, even if it wasn't php.
When using php-fpm you don't need the php module on every request as php-fpm is contacted as needed. This does increase latency and decrease performance for php requests, but the difference is indistinguishable for most people.
Slowloris attacks was/is one issue with non event-based webservers.
Both nginx and apache can handle a six digit number of php requests each second on a low end server. You application(Wordpress, Laravel) will without a doubt be a bottleneck before the webserver becomes the bottleneck.
But yeah, there was a lot of weird and misinformed reasons for why people recommended others to use nginx no matter what the use case was.
There's little reason to replace Apache completely, though.
Apache, nginx, caddy, Traefik, HAProxy, lighttpd, Varnish, Envoy are likely to be the major ones?
worker_processes auto;
(we also commonly see issues with buffering, and proxy host header not being set; the latter is an issue when code moves from an environment with physical hosts - which don't care about the host header - to virtual ones)
Otherwise if you have a listen directive for some host that doesn't have a default, this server will be used as the default for this particular listen directive.
For example doing `listen 80` and then requesting any non matching host will still return this server.
There is "Mistake 3: Not Enabling Keepalive Connections to Upstream Servers", where it says:
In the location{} block that forwards requests to an upstream group, include the following directives along with the proxy_pass directive:
proxy_http_version 1.1;
proxy_set_header "Connection" "";
Careless use of regexps in http routing is a common source of problems. Things like uri encoding, parameters as subdirectories and control characters is easy to get wrong. Treating user supplied headers as trusted is also easy to get wrong. It's not uncommon to see configurations which are remotely exploitable, which is another level of bad from what is described in the article.
I know of Apache, but that seems to be falling out of favour lately.
lately? more like for the last 10 years afaict.
The only thing that I don’t quite like is the logging format.
https://news.ycombinator.com/item?id=30440028
I'm starting to think Apache isn't bad once again after pulling several hairs out of nginx config complications over the last 10 years.
If I just wanted to alias system path of '/some/path/' as 'https://domain/another_path/' in the URL, I don't know how to do that... Is it 'alias' or 'root' and should it have a slash at the end or not??? I usually just give up and symlink the directory on the system.
Same with proxy_pass on ending slash that is confusing.
And the priority of 'location' block is a nightmare until you Google enough to figure it out every time I configure nginx, not to mention the cryptic convention of ~ or ~^ and the likes and the fact you can't effectively nest location blocks.
Being a Layer-7 load balancer, ALB is (almost) feature compatible with Nginx. You can do route based, header based load balancing, terminate SSL and so on.
For an internet facing app my preferred setup now a days is [CloudFront → ALB → application].
https://reversebrain.github.io/2021/03/29/The-story-of-Nginx...
Is it still nginx?
I know caddy gets a lot of attention but always assumed it was for smaller/less-trafficked sites (maybe I'm wrong).
sad
You can bind it to a specific IP address though (and then, by extension, also to an interface, but you need to know the IP).
Some failover configurations come to mind, in particular the one where the box doesn’t get an IP address on the said interface until the failover kicks in, but I don’t know why would anyone run this kind of a setup without NAT.
I have five disparate IP subnets whose firewall must NOW have to ascertain that ONLY a certain subnet can access this unbinded netdev … in case the NGINX (or any daemon) starts to sniff unwarranted IP subnet traffic.
And yes, you can bind to a dynamic IP assigned netdev.
And yes , I’ve had to replace NGINX with a web server that can do a netdev bind.
For example, currently i run my own Nginx ingress with Docker Swarm:
service A / B / C <--> Nginx <--> browser
server {
listen 80;
server_name app.test.my-company.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl http2;
server_name app.test.my-company.com;
# Configuration for SSL/TLS certificates
ssl_certificate /app/certs/test.my-company.com.crt;
ssl_certificate_key /app/certs/test.my-company.com.key;
# Proxy headers
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Reverse proxy
location / {
resolver 127.0.0.11 valid=30s; # Docker DNS
proxy_pass http://my-app:8080/;
proxy_redirect default;
}
}
Wrong. Nginx does throw the error and the request fails, but the ENTIRE web server goes down:
nginx: [emerg] host not found in upstream "my-app"
But fear not, supposedly there's a solution!
# Reverse proxy
location / {
resolver 127.0.0.11 valid=30s; # Docker DNS
set $proxy_server my-app;
proxy_pass http://$proxy_server:8080/;
proxy_redirect default;
}
nginx: [emerg] "proxy_redirect default" cannot be used with "proxy_pass" directive with variables
In summary, it's nice that there are workaround for problems like that, but it's a bit odd that the web server is otherwise so quick to fall over and die, as well as the fact that the defaults aren't oriented towards maximum reliability and resilience. That said, i still do like Nginx and actually wrote about how to use it as a reverse proxy on my blog a while back: https://blog.kronis.dev/tutorials/how-to-use-nginx-to-proxy-...
But at the same time, in situations where load isn't high enough for the web server to be the bottleneck, it's also helpful to evaluate the alternatives, notably: Apache2/httpd, Caddy and Traefik. It was actually pretty recently that Apache2/httpd got mod_md, which allows provisioning Let's Encrypt certificates out of the box, which is certainly promising: https://httpd.apache.org/docs/trunk/mod/mod_md.html
Or rather you can but apparently you need the expensive version of nginx to be able to have upstream servers that resolve hostnames sensibly.
What? That's not true. From the Caddy docs [1]:
> Caddy does its best to continue if errors occur with certificate management. By default, certificate management is performed in the background. This means it will not block startup or slow down your sites. However, it also means that the server will be running even before all certificates are available. Running in the background allows Caddy to retry with exponential backoff over a long period of time.
Caddy will actually keep your site online when other servers don't. See the bottom of this page: https://caddyserver.com/v2
TL;DR: if you run nginx I assume you want to know about these types of potential problems, so sticking it into the security database makes a lot of sense in my opinion for discoverability.
Simpler to set up for a basic proxy.
For more complex usage though, nginx is certainly outclassed (at least the OSS version)
Also if you're using Nginx everywhere else already, you might want to use a product you're comfortable with, even in the cases where Envoy could work for you. E.g. we use Apache almost everywhere, because performance is "good enough", and it allows us to use the same product for a lot of different client, due to the large feature set.
envoy is a L7 proxy focused on full duplex networking across multiple protocols with transparency and observability. It can also handle front/edge tasks although the configuration system is a magnitude more complex than nginx and it's also missing the app server abilities.
If you use envoy already then it might be easier to deploy it to the edge as well, and it's often used that way in Kubernetes as both service-mesh and ingress - but if you just need typical reverse-proxy/app serving than nginx is still far simpler to setup and maintain.