Hacker News Reader

Comment by lmm | Hacker News Reader

I think this post gets the complexity situation backwards. Sure, you can use a different protocol between your load balancer and your application and it won't do too much harm. But you're adding an extra protocol that you have to understand, for no real benefit.

(Also, why do you even want a load balancer/reverse proxy, unless your application language sucks? The article says it "will also take care of serving static assets, normalize inbound requests, and also probably fend off at least some malicious actors", but frankly your HTTP library should already be doing all of those. Adding that extra piece means more points of failure, more potential security vulnerabilities, and for what benefit?)

View on HN

> Sure, you can use a different protocol between your load balancer and your application and it won't do too much harm. But you're adding an extra protocol that you have to understand, for no real benefit.

Well, that depends...

At a certain scale (and arguably, not too many people will ever need to think about this), using UNIX sockets (instead of HTTP TCP) between the application and load balancer can be faster in some cases, as you don't go through the TCP stack...

> Also, why do you even want a load balancer/reverse proxy, unless your application language sucks?

Erm... failover... ability to do upgrades without any downtime... it's extra complexity yes, but it does have some benefits...

> At a certain scale (and arguably, not too many people will ever need to think about this), using UNIX sockets (instead of HTTP TCP) between the application and load balancer can be faster in some cases, as you don't go through the TCP stack...

Sure (although as far as I can see there's no reason you can't keep using HTTP for that). You can go even further and use shared memory (I work for a company that used Apache with Jk back in the day). But that's an argument for using a faster protocol because you're seeing a benefit from it, not an argument for using a slower protocol because you can't be bothered to implement the latest standard.

> using a slower protocol because you can't be bothered to implement the latest standard.

I thought we were discussing HTTP/2 but now you seem to be invoking HTTP/3? It's even faster indeed but brings a whole lot of baggage with it. Nice comparison point though: Do you want to add the complexity of HTTP/2 or HTTP/3 in your backend? (I don't.)

> I thought we were discussing HTTP/2 but now you seem to be invoking HTTP/3?

The article talks about HTTP/2 but I suspect they're applying the same logic to HTTP/3.

> Do you want to add the complexity of HTTP/2 or HTTP/3 in your backend? (I don't.)

I'd like to use the same protocol all the way through. I wouldn't want to implement any HTTP standard by hand (I could, but I wouldn't for a normal application), but I'd expect an established language to have a solid library implementation available.

> why do you even want a load balancer/reverse proxy, unless your application language sucks?

Most load balancer/reverse proxy applications also handle TLS. Security-conscious web application developers don't want TLS keys in their application processes. Even the varnish authors (varnish is a load balancer/caching reverse proxy) refused to integrate TLS support because of security concerns; despite being reverse-proxy authors, they didn't trust themselves to get it right.

An application can't load-balance itself very well. Either you roll your own load balancer as a separate layer of the application, which is reinventing the wheel, or you use an existing load balancer/reverse proxy.

Easier failover with fewer (ideally zero) dropped requests.

If the app language isn't compiled, having it serve static resources is almost certainly much slower than having a reverse proxy do it.

> Security-conscious web application developers don't want TLS keys in their application processes.

If your application is in a non-memory-safe language, sure (but why would you do that?). Otherwise I would think the risk is outweighed by the value of having your connections encrypted end-to-end. If your application process gets fully compromised then an attacker already controls it, by definition, so (given that modern TLS has perfect forward secrecy) I don't think you really gain anything by keeping the keys confidential at that point.

I write application servers for a living, mostly for Python but previously for other languages.

Nobody, nobody, writes application servers with the intent of having them exposed to the public internet. Even if they're completely memory safe, we don't do DOS protections like checking for reasonable header lengths, rewriting invalid header fields, dropping malicious requests, etc. Most application servers will still die to slowloris attacks. [1]

We don't do this because it's a performance hog and we assume you're already reverse proxying behind any responsible front-end server, which all implement these protections. We don't want to double up on that work. We implement the HTTP spec with as low overhead as possible, because we expect to have pipelined HTTP/1.1 connections from a load balancer or other reverse proxy.

Your application server, Gunicorn, Twisted, Uvicorn, whatever, does not want to be exposed to the public internet. Do not expose it to the public internet.

[1]: https://en.wikipedia.org/wiki/Slowloris_(cyber_attack)

As someone who designs load-balancer solutions for a living I cannot agree with this more.

I likewise assume that all servers are insecure, always, and we do not want them exposed without a sane load balancer layer.

Your server was probably not made to be exposed to the public internet. Do not expose it to the public internet.

> Nobody, nobody, writes application servers with the intent of having them exposed to the public internet

For rust, go, lua (via nginx openresty) and a few others this is a viable path. I probably wouldn't do it with node (or bun or deno), python, or similar but there are languages where in certain circumstances it is reasonable and might be better.

For Go, net/http is not something you should expose to the public internet, there's no secret sauce in there. It will just die to the first person to hit it with a slowloris or other DOS attack. Same with the common C++ options like boost.beast unless you're writing the logic yourself (but why bother? Just reverse proxy).

I'm unfamiliar with the common rust frameworks for http, but find it unlikely the situation is very different.

boost.beast also only talks HTTP/1.1 AFAIK, so if you want HTTP/2 or /3 on your frontend then you must put it behind a reverse proxy anyway.

> We don't do this because it's a performance hog and we assume you're already reverse proxying behind any responsible front-end server

What application servers have you written? I have never seen an application server readme say DON'T EXPOSE DIRECTLY TO THE INTERNET, WE ASSUME YOU USE REVERSE PROXY.

Most of them have a disclaimer in their deployment or tutorial docs, some with more strong language than others. Again, nothing bad happens if you don't, we don't write memory vulnerabilities into these servers. You are just far more vulnerable to DOS attacks.

* "We strongly recommend using Guincorn behind a proxy server" [1]

* "As a general rule, you probably want to: ... run behind Nginx for self-hosted deployments." [2]

* "A reverse proxy such as nginx or Apache httpd should be used in front of Waitress." [3]

For some, like uWSGI, they don't even want to talk HTTP (uWSGI supports its own protocol) and it's just assumed you're using a dedicated webserver to talk to public traffic. [4]

[1]: https://docs.gunicorn.org/en/latest/deploy.html

[2]: https://www.uvicorn.org/deployment/

[3]: https://flask.palletsprojects.com/en/stable/deploying/waitre...

[4]: https://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django...

Of course, don't expose to the public Internet.

Also don't expose plain text traffic to the internal corpnet where most attack originate from.

You use a reverse proxy because whenever you "deploy to prod", you'll be using one anyway, thus by not having TLS in your app, you had not built something you don't actually need.

Speculative execution cpu bugs. Or whatever the next class of problems is that can expose bits of process memory without software memory bugs.

That's already a fringe case. Do you really think everyone's writing web applications in a language like rust without any unsafe (or equivalent)?

> Also, why do you even want a load balancer/reverse proxy, unless your application language sucks

- To terminate SSL

- To have a security layer

- To load balance

- To have rewrite rules

- To have graceful updates

- ...

- To host multiple frontends, backends and/or APIs under one domain name

> backends and/or APIs under one domain name

On one IP, sure, for one domain you could use an API gateway.

API gateway is a fancy term for a configurable reverse proxy often bought as a service.

No, API gateway is a web service that has a main purpose of routing requests based on incoming requests.

Load Balancer maing purpose is to...balance the load across multiple backends.

Just because both can be implemented with a reverse proxy such as NGINX doesn't mean it's the same thing.

I said "API gateway is a fancy term for a configurable reverse proxy often bought as a service" and both "load balancer" and "API gateway" are common configurations of "configurable reverse proxy", often bought as a service.

Many load balancers have this functionality within them, even ones from years ago that aren't around anymore like Microsoft ISA/TMG. They're not web services, but they can route based on requests.

> To terminate SSL

To make sure that your connections can be snooped on over the LAN? Why is that a positive?

> To have a security layer

They usually do more harm than good in my experience.

> To load balance

Sure, if you're at the scale where you want/need that then you're getting some benefit from that. But that's something you can add in when it makes sense.

> To have rewrite rules > To have graceful updates

Again I would expect a HTTP library/framework to handle that.

> To make sure that your connections can be snooped on over the LAN? Why is that a positive?

No, to keep your app from having to deal with SSL. Internal network security is an issue, but sites that need multi-server architectures can't really be passing SSL traffic through to the application servers anyway, because SSL hides stuff that's needed for the load balancers to do their jobs. Many websites need load balancers for performance, but are not important enough to bother with the threat model of an internal network compromise (whether it's on the site owner's own LAN, or a bare metal or VPS hosting vlan).

> Sure, if you're at the scale where you want/need that then you're getting some benefit from that. But that's something you can add in when it makes sense.

So why not preface your initial claims by saying you trust the web app to be secure enough to handle SSL keys, and a single instance of the app can handle all your traffic, and you don't need high availability in failure/restart cases?

That would be a much better claim. It's still unlikely, because you don't control the internet. Putting your website behind Cloudflare buys you some decreased vigilance. A website that isn't too popular or attention-getting also reduces the risk. However, Russia and China exist (those are examples only, not an exclusive list of places malicious clients connect from).

> So why not preface your initial claims by saying you trust the web app to be secure enough to handle SSL keys, and a single instance of the app can handle all your traffic, and you don't need high availability in failure/restart cases?

Yeah, I phrased things badly, I was trying to push back on the idea that you should always put your app behind a load balancer even when it's a single instance on a single machine. Obviously there are use cases where a load balancer does add value.

(I do think ordinary webapps should be able to gracefully reload/restart without losing connections, it really isn't so hard, someone just has to make the effort to code the feature in the library/framework and that's a one-off cost)

> > To terminate SSL

> To make sure that your connections can be snooped on over the LAN? Why is that a positive?

Usually your "LAN" uses whole link encryption, so that whatever is accessed in your private infrastructure network is encrypted (being postgres, NFS, HTTP, etc). If that is not the case, then you have to configure encryption at each service level, which is both error prone, time consuming, and not always possible. If that is not case then you can have internal SSL certificates for the traffic between RP and workers, workers and postgres, etc.

Also you don't want your SSL server key to be accessible from business logic as much as possible, having an early termination and isolated workers achieves that.

Also, you generally have workers access private resources, which you don't want exposed on your actual termination point. It's just much better to have a public termination point RP with a private iface sending requests to workers living in a private subnet accessing private resources.

> > To have a security layer

> They usually do more harm than good in my experience.

Right, maybe you should detail your experience, as your comments don't really tell much.

> To have rewrite rules

> To have graceful updates

> > Again I would expect a HTTP library/framework to handle that.

HTTP frameworks handle routing _for themselves_, this is not the same as rewrite rules which are often used to glue multiple heterogeneous parts together.

HTTP frameworks are not handling all the possible rewriting and gluing for the very reason that it's not a good idea to do it at the logic framework level.

As for graceful updates, there's a chicken and egg problem to solve. You want graceful update between multiple versions of your own code / framework. How could that work without a third party balancing old / new requests to the new workers one at a time.

You terminate SSL as close to the user as possible, because that round trip time is greatly going to affect the user experience. What you do between your load balancer and application servers is up to you, (read: should still be encrypted) but terminating SSL asap is about user experience.

> You terminate SSL as close to the user as possible, because that round trip time is greatly going to affect the user experience. What you do between your load balancer and application servers is up to you, (read: should still be encrypted) but terminating SSL asap is about user experience.

That makes no sense. The latency from your load balancer to your application server should be a tiny fraction of the latency from the user to the load balancer (unless we're talking about some kind of edge deployment, but at that point it's not a load balancer but some kind of smart proxy), and the load balancer decrypting and re-encrypting almost certainly adds more latency compared to just making a straight connection from the user to the application server.

Say your application and database are in the US West and you want to serve traffic to EU or AUS, or even US East. Then you want to terminate TCP and TLS in those regions to cut down on handshake latency, slow start time, etc. Your reverse proxy can then use persistent TLS connections back to the origin so that those connection startup costs are amortized away. Something like nginx can pretty easily proxy like 10+ Gb/s of traffic and 10s of thousands of requests per second on a couple low power cores, so it's relatively cheap to do this.

Lots of application frameworks also just don't bother to have a super high performance path for static/cached assets because there's off-the-shelf software that does that already: caching reverse proxies.

It depends on your deployment and where your database and app servers and POPs are. If your load balancer is right next to your application server; is right next to your database, you're right. And it's fair to point out that most people have that kind of deployment. However there are some companies, like Google, that have enough of a presence that the L7 load balancer/smart proxy/whatever you want to call it is way closer to you, Internet-geographically, than the application server or the database. For their use case and configuration, your "almost certainly" isn't what was seen emperically.

You usually re-encrypt your traffic after the GW, either by using an internal PKI and TLS or some kind of encapsulation (IPSEC, etc).

Security and availability requirements might vary, so much to argue about. Usually you have some kind of 3rd party service you want to hide, control CORS, Cache-Control, etc headers uniformly, etc. If you are fine with 5-30 minutes of outage (or until someone notices and manually restores service), then of course you don’t need to load balance. But you can imagine this not being the case at most companies.

Tell me you never built an infrastructure without telling me you never built an infrastructure

The point being that all the code on the stack is not necessarily yours

I've built infrastructure. Indeed I've built infrastructure exactly like this, precisely because maintaining encryption all the way to the application server was a security requirement (this was a system that involved credit card information). It worked well.

Load balancers are nice to have if you want to move traffic from one machine to another. Which sometimes needs to happen even if your application language doesn't suck and you can hotload your changes... You may still need to manage hardware changes, and a load balancer can be nice for that.

DNS is usable, but some clients and recursive resolvers like to cache results for way beyond the TTL provided.

C fast, the rest slow. You don't want to serve static assets in non-C.

If you call sendfile with kTLS I imagine it'd be fast in any language

Answers from the article - the "extra" protocol is just HTTP/1.1 and the reason for a load balancer is the ability to have multiple servers:

> But also the complexity of deployment. HTTP/2 is fully encrypted, so you need all your application servers to have a key and certificate, that’s not insurmountable, but is an extra hassle compared to just using HTTP/1.1, unless of course for some reasons you are required to use only encrypted connections even over LAN.

> So unless you are deploying to a single machine, hence don’t have a load balancer, bringing HTTP/2 all the way to the Ruby app server is significantly complexifying your infrastructure for little benefit.

I've deployed h2c (cleartext) in many applications. No tls complexity needed

Good to know - neither the parent nor the article mention this. h2c seems to have limited support by tooling (e.g. browsers, curl), which is a bit discouraging.

EDIT: Based on the HTTP/2 FAQ, pure h2c is not allowed in the standard as it requires you to implement some HTTP/1.1 upgrade functionality: https://http2.github.io/faq/#can-i-implement-http2-without-i...

Why do you think that curl doesn't support h2c ?

It does. Just use `--http2` or `--http2-prior-knowledge`, curl deduce the clear or not clear by `http` or `https` URL protocol prefix (clear the default).

I said limited support and gave curl as an example because curl --http2 sends a HTTP/1.1 upgrade request first so fails in a purely HTTP/2 environment.

Thanks for bringing up --http2-prior-knowledge as a solution!

Reply on news.ycombinator.com