Comment by gafferongames | Hacker News Reader

Comment by gafferongames | Hacker News Reader

Parent Full thread

> _Throughpout_ is not problematic at all for the Linux network stack, even at 100gbps. What is problematic is >10gbps line rate. In other words, unless you're receiving 10gbps unshaped UDP small datagrams at line rate, the problem is non existant. Considering internet is 99% fat TCP packets, this sentence is completely absurd.

But games are not 99% fat TCP packets.

Games are typically networked with UDP small datagrams sent at high rates for most recent state or inputs, with custom protocols built on top of UDP to avoid TCP head of line blocking. Packet send rates per-client can often exceed 60HZ, especially when games tie client packet send rate to the display frequency, eg. Valve and Apex Legends network models.

Now imagine you have many thousands of players and you can see that the problem does indeed exist. If not for current games, for future games and metaverse applications when we start to scale up the player counts from the typical 16, 32 or 64 players per-server instance, and try to merge something like FPS techniques with the scale of MMOs, which is actively something I'm actually doing.

XDP/eBPF is a useful set of technologies for people who develop multiplayer games with custom UDP based protocols. You'll see a lot more usage of this in the future moving forward, as player counts increase for multiplayer games, and other metaverse-like experiences.

Best wishes

- Glenn

A modern server class machine can push "100Gbps" through the entire Linux stack just fine. TCP or UDP. With standard packet sizes (e.g. 1500 bytes.) We do this where I work. Yes a long time ago 1Gbps was hard, you needed jumbo frames, then 10Gbps was hard, then 100Gbps was hard. Right now where we are seems to be you don't need to do a kernel bypass unless you're running multiple 100Gbps NICs - that's an application where I've seen DPDK used in the wild.

EDIT: Might be some benefits in terms of latency through the stack and resource usage on the machine... But I don't think 10Gbps is where the pain is for these either.

> A modern server class machine can push "100Gbps" through the entire Linux stack just fine.

Maybe we'd like to use some of that CPU to run the game instead of pushing packets.

Maybe if we can push the packets more efficiently, we save $$$.

Maybe game servers get DDoS'd and it's great to be able to quickly drop packets without any linux kernel overhead.

Maybe I would like my games to not access my kernel?

Maybe I would like my games to not bypass my firewall and VPN?

Also, eBPF is not a way to shave CPU cycles, you would use DPDK/Netmap/BSD BPF for that, it's a way to lower latency. And we're talking single digit microsecond latency, where a typical internet link has millisecond jitter.

The use case is for dedicated game servers controlled by the developer running in data centers, not clients running on player's home computers.

If we're talking server tech then why does the article claim that 10gbps is becoming retail commodity? 10,40 and 100gbps is already the baseline for servers since a decade, I have not seen a server with 1gbps NICs for anything else than remote management or off band communication in a decade.

If it's for servers then the article still doesn't make any sense. See my other comment https://news.ycombinator.com/item?id=39939876

CableLabs.com, the creators of the DOCSIS standard for cable modems are part of an industry wide push towards 10G internet for ISPs. According to Neilsen's law, we should see 10G internet become more common than 1G internet is today within 10 years.

https://www.cablelabs.com/10g https://www.nngroup.com/articles/law-of-bandwidth/

You're mixing server and client use cases. And you're looking at a gaussian distribution. And DOCSIS is a shared medium.

Most of the world doesn't even have 100M internet: https://www.speedtest.net/global-index

By the time your game can use even 1G per client, XDP won't even be helpful for 100G. (Which, depending on use case, it already isn't.)

I repeat:

According to Neilsen's law, we should see 10G internet become more common than 1G internet is today within 10 years.

That's a cool thing. Let's look forward to it, and the rising tide that lifts all boats. The world will be a much more connected place 10 years from now, and the 10G target from CableLabs.com as well as DSCP packet tagging, L4S and other technologies will make the internet a lot better than it is today.

Why the negativity. Embrace the future :)

What do you mean by wanting games not to access the kernel? You don't want them making system calls full stop?

> First, DPDK allows reinjecting packets in the Linux network stack. That is called queue splitting,is done by the NIC, and can be trivially achieved using e.g. the bifurcated driver.

My apologies, the last time I looked at DPDK was in 2016, and I don't believe this was true back then. Either way, it seems that XDP/eBPF is much easier to use. YMMV!

No harm taken, kernel bypass technologies have moved fast in the last decade.

As a side note though, I would encourage you to revisit DPDK. I've worked in the low latency space for a long time, and used pretty much every solution out there, open source or proprietary, and DPDK is one of my favorite.

This is because DPDK is not _just_ poll mode drivers, it's a a full featured SDK for low latency packet processing. You get top notch thread safe memory pools to store packets, thread safe queues, hand written intrinsics optimized hashing and CRC algos, a cooperative scheduler in case you isolate your CPUs, etc, etc.

That sounds really good. To be honest, just not having to deal with the BPF verifier would be a plus.

> Packet send rates per-client can often exceed 60HZ

You do realize the most random Python program, written by the most random programmer, running on the most random computer, could easily read 10k TCP packets per second per core?

I absolutely do. Now let's see that random Python program handle 1M connected players each sending and receiving 1384 byte UDP packets at 244HZ, while also running game logic and simulation in the background.

Reply on news.ycombinator.com