HTTP/3 needs us to make firewall changes | Hacker News Reader

HTTP/3 needs us to make firewall changes

utcc.utoronto.ca

59 pointszdw67 comments

I think whitelisting outbound traffic based on port numbers is just a bad idea in general. Port numbers are arbitrary. Just because a protocol defaults to one doesn't mean it will always use it, and even if it requires one the use of a port number doesn't mean a specific protocol is being used.

I remember in university having trouble reaching several (admittedly niche) services, including some of my own, because they only allowed traffic to very specific ports like 80 and 443. I couldn't connect through SSH to my own servers because I applied the best practice of not having SSH run on port 22.

I ended up researching eduroam policies (which suggest as little blocking as possible) and arguing with IT about it for a few weeks, but the end result was just a few extra unblocked ports for protocols I could convince them to allow. So I moved all my own stuff to those ports, ignoring the protocols.

Perhaps they were trying to block (encrypted) P2P downloads and gaming. In such case it would be a "good enough" solution, because it is hard to convenience every each P2P peer or game server to use whitelisted ports.

Detecting protocol is hard, and much expensive than just port blocking. So I guess the admins were trying to do the best job possible with the limited equipment on their hands.

Detecting bittorrent isn't that hard, and neither is detecting most popular p2p traffic. It's CPU intensive though, requiring something like snort to run basic protocol heuristics on every packet. Just blocking as much outgoing udp as possible would be the poor man's solution to a network policy like that. There's also ways to cloak network traffic, of coarse.

Of course, in practice, all of the bandwidth can still be sucked up by p2p through VPNs which are probably allowed through the network, so I'm not really sure what the problem those restrictions try to solve is.

We just ran our own servers on port 443 - We tried 80 at first but I guess they did have something rejecting non-http traffic on that port, while I guess for 443 they just assumed everything not understood was https.

Also LoL worked whatever they did. Maybe that was just the admin's game of choice

I think my classmates in IT school learned much more from bypassing the network restrictions to play WoW in class than from the teachers themself.

Back when I faced similar issues SSHing to my home network from school and work, I forwarded port 443 external to 22 internal.

Very few people would be so silly as to close outbound port 443, and everyone expects that port to see encrypted traffic anyway, so it seemed like the logical choice.

It makes sense on a network with only managed servers - couple the firewall deny in with flow logs and forward to your monitoring system, it’ll alert you to suspicious traffic.

However on a general purpose network I agree. If you’re not using the outbound blocks to alert on you’re just inconveniencing someone who can work around it.

I have WireGuard endpoints on several ports including 53 and 443 to maximise the chances of getting through

Layer 4 doesn’t have any way to understand layer 7 though without some sort of deep packet inspection. That’s outside the realm being reasonable in a lot of cases.

I'm in an academic department and we've historically blocked outgoing traffic for two separate reasons. First, we consider some protocols actively unsafe to use because they transmit credentials in the clear, and we didn't want our users to accidentally do that. Telnet? Rlogin? Sorry, no. Second, we're in an academic environment where the attacker may be inside our network and poking at someone else's, so we want to shield outside parties from bad traffic we may be generating. In both cases, we're responsive to our users; if someone says 'I need this port', we'll allow the traffic (although we try to be selective about the destination).

(The third answer is that when we set up firewalls in the beginning, we consciously decided to start with a 'default block' policy.)

(I'm the author of the linked-to entry.)

"${name_of_new_technology} needs us (and other people) to make ${infrastructure} changes."

I don't really understand what the fuss is about. If your city builds a new road, you may need to take different turns to drive to work. If Python deprecates a function, you need to patch up your code. Just deal with it.

“Just deal with it” is problematic for adoption. Python being an excellent example, looking at Python 2.7. IPv6 being another one.

If firewall changes are necessary for HTTP/3 to be adopted (eg connection tracking in routers may not work anymore), that can cause big problems in adoption.

I don't think you can compare it to IPv6. IPv6 requires explicit adoption on the network side. In the case of HTTP/3, the vast majority of clients are already able to use it perfectly fine. They won't have any network firewalls in place that block UDP port 443. There are (virtually) no consumer ISPs that do that.

The point you are trying to make is that people are irrational, therefore:

Even if the effort required to adopt something would bring in disproportionate returns, they would skip it because the effort is nonzero?

(just a generalization, no idea if H3 worth it or not)

The effort required may be less than the benefit as a whole, but as an individual probably not.

Moving from Python 2.7 to 3 involves a lot of work for some people with an old stable code base, and benefits those writing new code.

Of course as Python is open there’s nothing stoping you from maintaining 2.7 yourself - except that’s more work, and until now you’ve likely benefitted from the work others have done more than you’ve contributed.

I had some fairly new code written in nodejs recently that fubared when node was upgraded. Looking at the tangled web of code to try to get working again, and the fact it was under 13 months old before it was reduced to a valueless mess, I spent 4 hours replacing its function with a stable language as that was a better use of my time.

> If firewall changes are necessary for HTTP/3 to be adopted (eg connection tracking in routers may not work anymore), that can cause big problems in adoption.

If firewall changes are necessary for HTTP/3, then it's a slightly 'dumb' protocol in some ways.

If that's a prerequisite, then perhaps more effort should have been spent in getting SCTP and DCCP accepted and run HTTP over one/both of those. QUIC seems to have basically the same feature set AFAICT.

Besides the architecture decision of having QUIC being on top of UDP, are there specific features that are much better than SCTP and/or DCCP? Are the other two better in particular areas? How complementary or competitive are the two approaches? (Genuinely curious.)

Firewall changes are only needed if your firewall rules were unreasonable to begin with.

> connection tracking in routers may not work anymore

What do you mean by this? Tracking of UDP flows has been supported by routers for decades.

Some people get it into their heads that just because their code works right now that it is somehow "done" and will never need maintenance, so they don't budget for any maintenance work or knowledge upkeep. Then when the world inevitably changes around the program, some of the initial assumptions are no longer valid and (possibly extensive) changes are required but no resources available to enact them. The only cheap solution available is to complain about the changes in the world and try to get them rolled back.

I feel like blaming developers (or management) for this is just fighting human nature which never ends well. Rather we should look at deeper root causes, like why these initial assumptions keep changing, and why there is so much churn in some ecosystems.

If the incentives aren’t there to do proper maintenance, no amount of blaming people for following those incentives will fix the problem.

I don't mean to blame anyone here, it is like you say: just human nature. The same is true about initial assumptions and software ecosystems btw, change has been their nature since basically forever and calling that a "root cause" seems like being angry at a thistle for having thorns. The world/human society has never been static, why should software be an exception to this?

In any case it does not really matter; either Utoronto will eventually rewrite their firewall configs, or they won't and then they will not be able to use HTTP3. These issues have a way of eventually becoming urgent enough to fix and at that time the incentives will sort themselves out.

> I don't really understand what the fuss is about.

There are only so many hours in the day/week, and it's usually already filled with work that needs to be done for internal needs. Someone external comes along unbidden and wants to add to your work for something that seems to have very little benefit for the organization.

Why should I do the work for them? What problem of mine does this effort solve?

My feeling (and why I wrote the article) is that it's easy to overlook that you need to do something active to enable your users to use HTTP/3. Previous upgrades to HTTPS of various forms generally required no firewall changes, unless you had a broken firewall that insisted on seeing specific things in TLS handshakes and dropped the connection otherwise. For instance, you can add HTTP/2 to your own servers or talk to HTTP/2 servers outside with no firewall change. But any shift to (or toward) HTTP/3 now requires firewall changes unless you're already passing UDP 443. You can't just deploy it on your own servers or assume that your users will be able to transparently take advantage of it; now you may need to do something active to enable it. It's easy to overlook that, especially since we're not likely to see active failures when HTTP/3 doesn't work, just a quietly degraded experience.

(I'm the author of the linked-to entry, and I should have done a better job of explaining this in the original entry.)

> you need to do something active to enable your users to use HTTP/3

Another way of looking at it would be that you have been actively blocking HTTP/3 (and probably many other useful but uncommon protocols) for your users and all that needs to happen to make it work is for you to stop doing that. Blocking all unknown things by default is just another form of protocol ossification.

That said when you're responsible for network security I can imagine how a block-by-default policy is tempting and hopefully people subjecting their users to such a policy will read your article and add an exception.

As pointed out elsewhere in thread, if your users are untrusted and not under your direct administrative control (e.g. students in a dorm, hotel guests, et c) then default-allow-everything jeopardizes your upstream transit connection (e.g. if they start spamming).

You owe it to others on the internet to not allow everything out indiscriminately from untrusted strangers who happen to be on your access network. It’s just neighborly to make sure you’re not sending ddos or spam, for example.

> if your users are untrusted and not under your direct administrative control (e.g. students in a dorm, hotel guests, et c) then default-allow-everything jeopardizes your upstream transit connection (e.g. if they start spamming).

How is that different from an ISP? Or do you think ISPs should also block everything except TCP ports 80 and 443?

ISPs throttle and block lots of things. Many residential ISPs block port 25, for example.

I've had the unfortunate experience of running machines behind other people's corporate firewalls. Some corporate networking teams are a blessing to work with and I learn debugging tips from them on calls, others stick to archaic security theater.

Getting a standardized UDP egress port will be a godsend to anyone doing corporate IoT. You'll finally be able to connect your boxes to a VPN and remotely manage your hardware.

> I've had the unfortunate experience of running machines behind other people's corporate firewalls.

This is our bread & butter. We have to deal with many unique and extremely secure environments (banking).

99% of these sorts of problems can be eliminated by being up-front with your system requirements and expectations from your customer. We are in a position to fire our customers if they are not willing to work with us on critical technical constraints like this. You should not punish your technical team because your customer is unable to satisfy their part of the bargain, unless there is some special fee or other compensatory measure in the contract.

That said, we are also mindful of the impact & suffering that shiny new protocols may cause with arbitrary network middleware, even if the customer is willing to work with us on everything.

> 99% of these sorts of problems can be eliminated by being up-front with your system requirements and expectations from your customer

At my last job, and also every IoT company I've worked at, we've implemented an HTTP endpoint that just returned the current time in milliseconds. Why would we do this? Well, it turns out, many corporate networks just block NTP all together and refuse to open up egress on NTP because "UDP is insecure and we only allow HTTPS"

> We are in a position to fire our customers if they are not willing to work with us on critical technical constraints like this. You should not punish your technical team because your customer is unable to satisfy their part of the bargain, unless there is some special fee or other compensatory measure in the contract.

I would have loved to have this freedom but unfortunately smaller startups, that are just getting established, don't always have this ability. In the end this was solved in a few different ways. One of my favorite is to just get a 4G modem in your device and an IoT data plan from any telecom. You can also pay a little more to get the ISP to NAT you to a VPN you can talk to all of your devices directly.

I've never used this provider but you get an idea of the market: https://www.oliviawireless.com/ipsec-vpn-for-iot

This person has very strong opinions on a protocol of which they profess little knowledge.

They seem to think that HTTP/3 somehow 'handles' the fallback to TLS/TCP. When in reality the browser will fallback to an older HTTP protocol over TLS/TCP.

Adoption by their corporate network is not in any way important to the success of HTTP/3, in reality mobiles are the primary target market.

I understand a university lacking the IPv4 address space to allow arbitrary incoming traffic (although IPv6 can solve a large chunk of that problem), but why on earth would you block outgoing ports from a research institution? Perhaps you can block protocols like SMB or NetBIOS to prevent leaking passwords through the browser, but blocking any outgoing port just because?

There's also discussion of allowing port 443/udp as an incoming port but that's not necessary "because there are no stable servers yet". That's a terrible approach for a research institution, this university will always stay behind the curve because their IT department decides what is or isn't considered part of the "real" internet.

This is the kind of thinking that made DoH win over DoT: "I don't see a use for it and I'd be giving up control so screw you".

At my university, every device gets a fully reachable IPv4 and IPv6 address, optionally coupled with a subdomain. I've never heard anyone having trouble with this approach, probably in part because the network administrators do preemptive port scans to detect vulnerable devices. The biggest problem I have is that some gamer mouse software is broadcasting UDP packets over the entire network, ruining everyone's battery life with useless wakeup of the WiFi hardware.

> I understand a university lacking the IPv4 address space to allow arbitrary incoming traffic (although IPv6 can solve a large chunk of that problem), but why on earth would you block outgoing ports from a research institution?

Because production and research networks should be separate.

If there's IP network research going on, it should be isolated from your labs, printers, secretaries, etc.

* https://en.wikipedia.org/wiki/Science_DMZ_Network_Architectu...

> At my university, every device gets a fully reachable IPv4 and IPv6 address, optionally coupled with a subdomain.

Does that include all the Windows desktops in Finance? Your Oracle/SAP HR servers? Does each "device" you reference also include the SCADA stuff that Facilities uses to monitor environmentals?

There are plenty of devices at an university that should have outgoing (and incoming) filtering.

As far as I know, yes, everything is connected to the internet with publicly reachable IPs, although some departments are usually placed behind an extra firewall.

My university does do filtering, taking action against port scans, traces of active malware, even does some DNS introspection I'm not 100% on board with.

Those desktops in Finance and those servers should have decent firewalls on them anyway, locked down with software policies. An extra layer of firewalls against arbitrary incoming connections is probably also for the best. Outgoing connections, though? Only blocked to known malware domains as far as I know. There's a third party that does network upstream that further monitors the security so I don't know the details on that, but I do know (and have run into this myself) that a mere lookup of a malware domain will get your network connection jailed.

I don't know about any IoT crap in use for monitoring. That stuff should be on a completely separate network anyway, behind many layers of VLANs, firewalls and static routes, because the tech simply cannot be trusted. If you mix Facilities and IoT into one network, you're bound to have problems, no matter the great firewall you put around the network perimeter.

My experience is mostly from the student side of things, but the system for registering WiFi and ethernet devices is not different from the employee side of things outside the OAuth authentication flow.

> My experience is mostly from the student side of things

Yes, and the author of article is writing from IT side of things, and he has to deal with non-student things as well.

> Eventually we may also want to enable inbound UDP to port 443, so that people can run web servers that support HTTP/3.

Apropos inbound traffic, what is the approach regarding inbound replies?

As in: Even if you do an outbound request, you usually expext an acknowledgement or reply from the server, which constitutes inbound traffic.

With TCP, firewalls can distinguish between unprompted inbound packets and replies by tracking which TCP connection the packet belongs to. This is possible because TCP packets contain unencrypted connection information that can be read by firewalls.

With generic UDP, there is no reliable connection information, so firewalls have to rely on heuristics like hole punching [1] - which are less precise and suboptimal from a security perspective.

I believe QUIC has connection information, however it's encrypted, so it's not accessible to firewalls. So does that mean, the future is hole punching everywhere, or was this changed so the connection info is now public?

[1] https://en.m.wikipedia.org/wiki/Hole_punching_(networking)

NAT and stateful packet inspection (SPI) work with UDP similar to how they do with TCP, but depend a little more on heuristic timeouts because there is nothing equivalent to TCP flags.

Outbound packet 4-tuples (source IP, destination IP, source port,, destination port) are tracked and translated with NAT. For some heuristic amount of time after, if an incoming UDP packet is received matching the same 4-tuple it is let through and the translation is reversed. This is UDP "connection tracking", and the details are a little more complicated than described here but not a lot more.

As with TCP, these are considered outbound connections because the initiating packet is outbound. There's no need for hole punching, which is a set of unreliable techniques to enable inbound connections.

Firewalls and routers have supported NAT for UDP as long as they have for TCP in practice. They haven't supported NAT for other IP protocols such as native SCTP.

This a major reason why QUIC is built on top of UDP, so outbound QUIC connections work already over most NAT routers, at home, at work and mobile.

That clears up a lot. Thanks!

Do we need HTTP/3?

The google thinks so.

We don't need anything. Like any technology it has pros and cons. The pro side being it improves performance for users with certain network conditions.

No... it is just a nice to have for some large sites. For most HTTP/1.1 is still good enough

For unreliable networks we do need it. TCP has always performed poorly when there is a lot of packet loss.

My personal opinion would prefer to improve those networks vs going UDP for http(s). Until spoofing, amplification attacks on UDP are properly tackled and not causing most internet outages, I'm not really looking forward to a UDP web (http/3)

Does http3 have a problem with amplification on the first answer packet?

I'm not sure spoofing is something transport protocols have to solve though. Authenticated bgp and source filtering is the layer for it and we needed it for decades already.

If I remember the amplification is about 3x, you need acks for the server to send more than that.

> Until spoofing, amplification attacks on UDP are properly tackled

This is already addressed by QUIC (the transport protocol used by HTTP/3). Those things are no more of a problem for HTTP/3 than they are for HTTP/2 or HTTP/1.

Wasn’t that the whole reason behind improved congestion algorithms like bbr which no longer assumed minor packet loss was the sign of congestion?

That all depends on the congestion control algorithm. Different implementations of TCP can use different algorithms while still being compatible. The same is true of QUIC.

The recently published QUIC standard specifies a tweaked version of TCP NewReno. See https://www.rfc-editor.org/rfc/rfc9002.txt and https://datatracker.ietf.org/doc/html/rfc6582 But AFAIU this is not the algorithm Google itself uses for QUIC, nor the one they used for their famous benchmarks showing latency improvements on mobile.

To derive maximum benefit (or even just most of the benefit) of a tailored congestion control algorithm, you'll want to control both sides, keeping them in sync as you iterate improvements and changes. And maintain different flavors for different application environments. Only huge companies like Google and Facebook will be able to do this effectively.

QUIC does have more than just different congestion control For example it has forward error correction which can be very important for lossy connections.

There are other improvements that can't really be added to TCP either such as roaming between IP addresses. (MPTCP exists but IIUC requires make-before-break which is not always possible).

I have the opinion that we don't need anything past HTTP/1.1 on 3G internet. Anything better is a luxury which is not necessary. But that's my threshold of comfort.

How long does it take to fully load the average web page over 3G?

I don't know about average web pages, but websites that I use regularly are loading quite fast. 3G in my village is something around 10 Mbits/s, youtube is absolutely fine, for example.

PS may be it's not strictly 3G, but it's displayed as 3G on my phone.

According to Google, 3G is generally limited to 3Mbps in practice (7Mbps in theory) so I think you might be getting lucky

You would be right if 3G didn't jump down to 2G randomly, 4G does not have this flaw, 5G is meaningless.

HTTP/1.1 is the final transport for humanity, but you need fiber (external IP, preferably static, but operators/governements are not helping) to sell something.

HTTP/2 has head of line blocking issue.

HTTP/3 wont become a standard outside of biased actors like Google.

Http3 works or will work soon in major browsers and cdn's. It will cary a large chunk of internet traffic.

A lot of the world has killed off 2G (e.g. in Australia it all got shut down 3–5 years ago), and it’s on the chopping block in a lot of the rest of the world (e.g. looks like the remaining 2G services in the USA goes EOL by the end of next year). 3G is also already on the chopping block in many places (e.g. looks like next year in the USA, and in Australia its spectrum has steadily been being reallocated in recent years, and the biggest telco is scheduled to kill it altogether in 2024). Fortunately after that it’s designed in compatible ways, so 4G and 5G and whatever comes after that can all coexist.

For the HTTP versions, I’d put it more this way:

HTTP/1.1 is the baseline that will continue to work forever.

HTTP/2 is normally a significant improvement over HTTP/1.1, but it has one crucial flaw which makes it worse than the alternative of half a dozen HTTP/1.1 connections on connections with high packet loss.

HTTP/3 is the best of both worlds, roundly better than HTTP/1 and HTTP/2, but because it’s using UDP, it’ll have some teething issues for a few years because of badly-configured corporate equipment. But it will become widely supported in server software—there’s no reason for it not to.

I think most reasons for sequels is that people need something to do, HTTP/1.1 works for everything but people needs to feel they can improve things.

Unfortunately everything peaks because of physics and energy.

I'm not adding HTTP/2 or 3 to my HTTP app. server; HTTP/1.1 does everything I'll ever need and I have to focus on other things!

You're probably not serving millions of users every day though.

Btw, most production setups don't really have to do anything to support it, as pretty much all load balancers already do. And there isn't any big performance gain if it's only intranet traffic.

Luxury! Back in my time we had to wait for the mailman to bring us our webpages on printed out paper! That we had to read by the light of candles which we had to make ourselves! These young kids with their fancy "HTTP is a requirement" mindset.

/s of course. But what a wonderfully arbitrary threshold to set it at 3G+HTTP/1.1 :)

Speaking of which: Azure's default "RDP" firewall rule blocks UDP, even thought Windows Server can use UDP for RDP!

This is a common problem, where network firewall people think one application == precisely one port on one protocol.

The most egregious of these kinds of issue is with DNS rules specifying UDP only. Drives me up the wall :).

Because UDP is connectionless, you’ll also need to enable inbound UDP traffic for outbound requests. Or am I missing something?

Conntrack is pretty clever in the way it classifies "connections", it has heuristics to determine what is a "related" connection (with many manually written rules for many protocols). For UDP it decides that if some packets go one way with UDP, then packets that arrive the exact opposite way not long after are the same connection.

Most protocols that implement 1:1 sessions and use UDP encapsulation conform to a port usage convention that more or less works despite middeboxes.

Type dig www.google.com @1.1.1.1

Note how it works despite being udp out though your firewall, being batted to a public ip, and the reply on a connection less protocol coming back in and being let through and correctly sent to the correct internal ip, even though someone else in another ip did exactly the same for www.bing.com at the same time.

did the article possibly mix up outbound and inbound port?

yes, you need to enable udp inbound, outbound will be a random port.

Reply on news.ycombinator.com