Hacker News Reader

Comment by tothrowaway | Hacker News Reader

Can you share more details about what the 4 HAProxy servers are doing? I run a SaaS too, but on a single beefy machine handling web and DB stuff. I'm curious what I'm missing out on not using HAProxy in front.

View on HN

To answer the question of why four of them:

* the traffic from the monitored systems comes with spikes. Looking at netdata graphs, currently the baseline is 600 requests/s, but there is a 2000 requests/s spike every minute, and 4000 requests/s spike every 10 minutes.

* want to maintain redundancy and capacity even when a load balancer is removed from DNS rotation (due to network problems, or for upgrade)

There are spare resources on the servers, especially RAM, and I could pack more things on fewer hosts. But, with Hetzner prices, why bother? :-)

I understand having multiple proxies, but it seems a bit odd to have more proxy servers than web servers, no? Do they also do caching so that of the up-to 4,000 requests/second, a lot of them never reach the web servers? Presumably the Python code is more resource intensive than HAProxy.

I didn't realize you were getting that many requests per second. That's quite remarkable when you only have 18 app cores available. In my experience, you're lucky if you can get Django to service a request in 10ms. So I'd expect closer to 40 cores to handle that level of traffic. I suppose since it is spiky, and not a particularly latency sensitive application, you can get by with fewer cores.

I've got a small program written in Golang for handling the HTTP "ping" requests. There's also rate limiting, not every request makes it past NGINX.

Damn, I never thought about it, but I guess the next cron/scheduling system should really be configurable down to the second, and if it is not defined set a random value between 0-59 seconds as a default. That might make the entire internet traffic and global compute use less spiky.

Why uWSGI? Every time I've talked to someone who uses it and got them to try gunicorn they've seen quite big performance improvement for no code change and switched.

In the post he says they're for SSL termination and for rolling upgrades of the servers behind them.

Reply on news.ycombinator.com