The perils of the “real” client IP
adam-p.ca
adam-p.ca
The biggest offenders are internet of things backends for registered devices I "own" and streaming services I'm paying for.
Edit: Here's a concrete example: I looked up my CGNAT IP and saw that it was flagged as a malicious actor because someone ran a port scan from it a month ago.
Now that this offence has started to age out, a few services started working again. My entire ISP can be trivially DOS'ed with a raspberry pi and NMAP!
Of course, other services seem to just do per-IP rate limiting, so they run at << 1MB/s during peak hours. Fast.com and Speedtest.net claim the connection is healthy.
This is a problem with service. If the service doesn't want to fix it, then that sounds like a potential market opportunity, no? I'm new to thinking about the business side, but it seems like I want to remove any obstacle I can preventing a customer from using my service.
Where I live (US), most ISPs won't give you static IPv4 addresses unless you are on a "business" plan. Beyond that, you also have to pay for those static IP addresses (it's ~US$25/month for 5 static IPv4 addresses, with one of those wasted because I'm forced to use the ISP's router, even though I already have one).
Perhaps where you are, static IPv4 addresses are available on "consumer" plans, but here not so much.
and this is why IPv6 isn't coming anytime soon, at least not within the next few decades.
You read comments here on HN every time ipv6 comes up how it caused great issues for them and they turned it off, but unless they are on a really shitty small ISP, it's their own fault because they don't use the router from the ISP but had to build their own setup and messed something up.
As people switch ISPs or get replacement routers that have it enabled, the world gradually moves over, and nobody apart from the nerds takes notice.
> The biggest offenders are internet of things backends for registered devices I "own" and streaming services I'm paying for.
Wait those are the services that are rate-limiting you? They're not smart enough to rate-limit based on your account credentials?
Only if users complain enough. Often they complain to the reverse proxy host[0] or website itself[1], when it would be a solved problem if IPv6 were properly deployed further.
0: top result on google for 'Cloudflare blocked me' with 38k views https://community.cloudflare.com/t/cloudflare-is-blocking-me...
1: https://www.coursera.support/s/question/0D51U00003BlYiVSAV/y...
This works until a VPN node starts having tens of thousands of users from one IP address.
I ran into this problem 15 years ago or so when I built my own ipv6 tunnel for home use using a hetzner server. Couldn't edit on Wikipedia, asked why, and apparently they blocked a whole /48 because of spam. Then they argued the whole provider should be blocked because it's a "tunnel provider" (it isn't) at which point I didn't even want to argue anymore and just stopped editing.
It isn't so much that you need to do it that way, but that there is no more practical way despite the inherent problems. Which has effectively the same end result, but thinking that way highlights the fact that CGNAT and other IPv4 limit “solutions” cause as many problems as they solve.
Nothing "black hat" with that, at all.
...Unless you're referring to that those "streaming-friendly VPNs" are (acquiring residential IPs in sketchy ways from sketchy providers, such as compromised smart devices and other customers).
I can't believe how some companies assume a port scan is some sort of attack. I've actually gotten automated abuse emails back when port scanning some addresses. Madness.
I mean if you saw someone standing on your sidewalk taking note of all the unlocked doors and windows they see you would be right to be kinda suspicious. The over under on “they’re casing your house vs they’re a window lock enthusiast” isn’t great for the enthusiast. But unlike IRL you can just say, “ya know what I don’t really care to support any exceptions and so anyone who’s checking my windows is autobanned.”
Another problem that I suspect is common: connections that sometimes go through a reverse proxy, sometimes not. In my setup, the "not" ones in theory are trusted (on the LAN or localhost) but still it doesn't feel right to not really know if the header came from the proxy or not. I could add a shared secret or something to the header, but given that a LAN-based attacker could sniff everything (not just this but also the actual user credentials/traffic) via ARP spoofing or something, it probably doesn't as much sense to bother before getting rid of the plaintext backhaul.
Speaking of which, it'd be nice to make the TLS end-to-end: from the user through a proxy that doesn't decrypt all the way to the application server. But I'm not sure what the state of the art there is. It used to be possible to dispatch based on the SNI traffic and then proxy at the TCP level, but I know TLS 1.3 added encrypted SNI. Not sure if the proxy can force clients fall back to non-encrypted SNI. I could imagine the spec authors making a point of not allowing this so a man-in-the-middle can't find the intended host. Maybe the two legs just have to be encrypted separately now.
[1] http://nginx.org/en/docs/http/ngx_http_proxy_module.html#pro...
[2] https://docs.rs/http/0.2.6/http/header/struct.HeaderMap.html...
[3] https://blog.encrypt.me/2013/11/05/ssl-added-and-removed-her...
Encrypted SNI is not the default. You've got to do a fair bit of work to do it. You'd probably want your proxy to be able to decode it to direct the traffic anyway.
My boringproxy[0] project has a mode for doing SNI routing at the reverse proxy, and tunneling it all the way back to a local client machine which handles the actual TLS termination. The client also gets certs automatically from Let's Encrypt. So you end up with automated end-to-end encryption where the boringproxy server/VPS can't decrypt any of the traffic.
Early versions of boringproxy acted as a more traditional reverse proxy, terminating the TLS at the server then making a new HTTP request upstream. Eventually I added the ability to move the HTTP proxy into the client to enable e2ee. Most recently I implemented raw TLS all the way. There are tradeoffs:
Pros:
* End-to-end encryption.
* Simplicity. You just need to peek at the SNI and look up what TCP tunnel to pipe into.
* Things like WebSockets and other hop-by-hop requests don't need to be implemented by the proxy.
Cons:
* You lose the ability to do compression/caching/CDN/etc at the server.
* You can't support new protocols like HTTP/2, HTTP/3, etc at your server because it only understands TCP wrapped in TLS.
In practice, I think these tradeoffs are totally worth it for self-hosting. I've found performance to be great for my purposes.
As for encrypted SNI (ESNI, now wrapped into encrypted client hello, ECH), I'm pretty sure it will be implemented in a tiered approach like you surmise. So in my case the boringproxy server will have the keys to decrypt the client hello, but the origin servers will still control the actual TLS decryption.
One addition I'd suggest is to add going into the implications of enabling "client port preservation" with an AWS ALB. This can be a death sentence if you decide to turn it on without intimate knowledge on how all connected apps are trying to figure out the real IP. It appends the client port to the IP with a colon in X-Forwarded-For and if you happen to read IPs without expecting this then it can potentially cause IP values to appear invalid. It's super app dependent on what would happen but it has a lot of side effect potential.
As a result I had a long conversation with AWS where I told them it was ridiculous that ALBs allowed garbage to be inserted in the only header that contains this important information… suffice to say they did not care.
That's pretty bad. It makes the already perilous header even worse. I'll a note or addendum about it in the post when I get a chance.
(@kingforaday: You're welcome!)
I feel comfortable about disclosing this now since we have things like SciHub (which may have used this trick at one point), and they fixed it a few years ago.
Basically the first device in the infrastucture, whether it is a load balancer, a cache, etc. should set (or overwrite, if it's been spoofed) a custom proprietary header with the client IP. And that's what you can rely on - either it's set and you have the IP or its not and you can't really trust anything else.
He mentions Azure and a few others doing this and that sounds like the only correct approach.
(Although there's still an issue when the infrastructure is changed, and the application code is not updated - in that case the old header might no longer be set, and could now be spoofed)
The problem here is that your border box should remove any such header that comes from an untrusted network, and not append to them.
Minefields within minefields.
The systems involved took proper care that those headers and their values really came from us, and not some outside system. Of course, we also had some header akin to Our-Site-Prefix-Peer-IP-Addr, to know which TCP peer the outermost system was actually talking to. In combination with evaluating other headers, like the X-Forwarded-For that this article handles in delightful detail, there was a lot of interesting detective work to be had in terms of which clients tried to spoonfeed us which kind of would-be-spoofed data.
These days, Carrier-grade NAT and the slow death of the End-to-end principle make these techniques woefully inadequate to discern between "legitimate" use, and clients you'd rather keep out. Sadly.
Setting up an application server behind a reverse proxy to use the "real" client IP is unfortunately very typically just a trial-and-error based process, with very little room for this kind of nuanced security-conciousness, because the configuration and exact behavior is all so non-standardized across different implementations of reverse-proxies and application servers... Typically users will just try different configuration settings until they find a combination that seems to work, and you would actually need to dig in with curl and tshark to understand the edge cases, because the documentation of the application-specific implementation is typically just one brief sentence...
Getting XFF working correctly through a complicated HTTP stack with multiple layers of nginx/haproxy/apache proxies (yes, they have different non-overlapping feature sets), custom backends implementing custom XFF handling/forwarding, and jetty/spring backends upgraded across a major version bump that changed the implementation and configuration properties related to XFF handling was insanely difficult. And of course it broke when migrating from a F5 LB to an AWS ALB, because it behaved differently for that one edge-case for an important customer... highly recommended to just override the entire XFF header with a single value at the appropriate point in your stack, if at all possible.
If just the naive leftmost-first vs rightmost-ish-with-configurable-list-of-trusted-upstream-proxies wasn't enough, then yeah, HAProxy does the thing where it adds a new 100% standards-compliant header continuation line [1] that maybe 1% of backend application developers have ever tested with. And trying to configure HAProxy to interpret the incoming XFF headers for logging/access-control ~is~/was even more weird [2].
[1] https://github.com/haproxy/haproxy/issues/44 [2] https://github.com/haproxy/haproxy/issues/90
This is very true, and I narrowly missed getting burned myself by thinking, "looks good to me!" after seeing my own IP.
However... I take a dig in the conclusions at the implementers of security-related libraries. I don't think it's okay for them to stop at "seems to work". They should be taking the time to fully understand the problem space.
> highly recommended to just override the entire XFF header with a single value at the appropriate point in your stack, if at all possible
I agree, and probably should have emphasized that approach. (And maybe will, and will definitely add a note at the end.) I didn't really give any/enough attention to configuring your first proxy with a custom single-IP header. (Partly because I was writing more for people who are trying to use what's available.)
My TL;DR on the topic is:
1. There's no single correct mechanism for determining the client IP address. Any standards and headers you may have come across are merely hints. Your solution has to be custom-built taking into account your use case, your network topology and the service provider(s) you are reliant on. If any of the above changes, your code has to change as well.
2. There is no single "correct" answer for client IP address, no matter how much effort you put into 1. Any value you come up with is at best an approximation, so treat it as such, especially if you are using the value for something critical like authorization.
https://voidnet.tech/chaos/blog/do-not-trust-x-forwarded-for...
But, if you don't trust any headers at all, it's tough to do much with HTTP.
I personally prefer that the load balancer speak "proxy protocol" to the front proxy, and then the front proxy simply make a decision as to what the external IP is and relay that to upstream applications with a proprietary header (x-envoy-external-address). Upstream applications that really want to be careful about the external address then verify the signature of envoy's TLS certificate when the connection comes in. (I use some mTLS for my homelab stuff, but don't have any software that cares to check this. I use Envoy's rate limiter rather than application specific rate limiting, and use Envoy's logs to get the IP address and the x-b3-trace-id header to correlate access logs with application logs.)
Proxy protocol is regrettably problematic from time to time, however. By default, Kubernetes tries to be "smart" about routing. If you have a LoadBalancer with internal IP 10.123.234.56 and external IP 54.43.32.21, traffic originating in the cluster with a destination address 54.43.32.21 does not transit the cloud provider's load balancer, it just gets rewritten to the internal IP address 10.123.234.56. This means that it doesn't use the proxy protocol, and the front proxy rejects the connection. This is generally not a problem, but will be annoying if you host an internal container registry in-cluster, because containers will be named for the external container registry and will thus resolve to the external IP address. The connection will skip your cloud provider's load balancer which adds the proxy header, and your front proxy will reject the connection, causing failing pulls. Most people will never hit this, because their cloud provider just provides a container registry (when I set up all my container stuff, my cloud provider of choice didn't offer this; they do now). But if you ever think internal applications will intend to use the public Internet to connect to other internal applications, watch out. This can come up if you do OIDC things internally, for example. (Apps will want to grab the .well-known configuration keys and JWKS key material from the external address that it will eventually redirect clients to. You will be confused when the configuration fails to load.)
In conclusion, what a mess.