Hacker News Reader

Comment by FBISurveillance | Hacker News Reader

Thanks for the post!

Is there anything you could suggest looking into to improve nginx processing of high volume large POST bodies (5KB, tens of thousands per second)?

I'm using kernel 4.13 with BBR congestion control on 20Gbps network and seem to hit this weird bottleneck when it does not matter how many nginx processes I have, it works similarly terrible on both 16-core and 64-core servers. (Of course irq/process affinity is in place which makes me think it's nginx issue.)

View on HN

You mention 5KB...is there some chance that it exceeds 8KB?

There's a setting called client_body_buffersize. (http://nginx.org/en/docs/http/ngx_http_core_module.html#clie...) It defaults to 8k on x86-64.

If you exceed that setting, it writes the body to a temporary file, which slows things down.

Maybe try bumping that up to 16k?

Also see the other related settings on that page.

Edit: Maybe also test out switching from urlencoded to multipart for the post? For some types of data, urlencoded can bloat the size quite a lot.

Not an nginx user here, but I do know sometimes certain modules in Apache will add significant latency on POST payloads by waiting for the entire POST payload to complete before sending to the backend (in a reverse proxy setup). The idea is that if the backend fails, it can retry on the next node. For large payloads this sucks for latency.. No idea if nginx has this problem.

This is a configurable option in nginx. You can have it wait until the entire request is recevied before passing to the upstream (backend) or have it stream as data arrives. I seem to recall there is still a manageable buffer when streaming too, though it's been a few years since I looked at that in detail.

At that speed it might not be Nginx. Check out Cloudflare's other blog posts too.

https://blog.cloudflare.com/how-to-receive-a-million-packets...

And thanks OP. I'm sure thousands of companies do these kinds of optimizations but CloudFlare is one of the only to bother sharing.

Reply on news.ycombinator.com