Hacker News Reader

Comment by personZ | Hacker News Reader

The disparity between what the OS can do and what the hardware is capable of delivering is off by a few orders of magnitude right now. It's downright ridiculous how much performance we're giving up for supposed "convenience" today.

What are they talking about?

View on HN

The operating system APIs (sockets) basically cannot delivery to/from userspace anything like what the hardware can do. The classic example is 10Gb ethernet, where if you want to do something like packet switching, you cannot do anything like line rate, due to cost of context switches, memory copies, scheduling, cache misses in the stack layers and so on.

Given that servers are getting 10Gb, 40Gb or more, this is becoming more and more of an issue. The hardware is capable of it, but the abstractions are getting in the way.

There is an OS overhead for sure (although you can mitigate that with things like Zero Copy solutions), but it generally imposes a couple of percentage points of overhead. I was thrown off by the several orders of magnitude claim.

Reading it again, it sounds more holistic -- that a simplified, monolithic application with a local, lightweight database is much higher performance than, for instance, an n-tier, distributed database type solution.

It is not just a few percent for applications where userspace needs to look at each packet. There are some performance comparisons on the netmap page for example http://info.iet.unipi.it/~luigi/netmap/

That comparison is of questionable merit. It compares bulk, unmetered packet generation (in an OS, worth noting, just as a kernel module) against netsend, which is intentionally a rate limited generator (using busy waits, as an aside, which means that the CPU probably was at 100%...doing nothing), with the significant overhead that entails.

where userspace needs to look at each packet

That benchmark is like those naive "look how fast my web server is when it returns just status code 200" comments. Even if we accepted that the overhead was anywhere close to the linked, which it isn't, the moment you actually do something with the packets those savings disappear into rounding errors.

netsend and other apps used in that comparison were not rate limited. Surely that test only measures system's overheads, but i put that disclaimer very clearly in all papers and talks (btw "which it isn't" suggests that you have different numbers so please let us know). Surely, sometimes these per-packet savings are irrelevant, but there are a number of use cases where this kind of savings matter a lot. This is true for netmap, dpdk, DNA and all other network-stack-bypass frameworks.

On passing, people typically use the term 'os-bypass' but netmap relies on the OS for protection, synchronization, memory management etc -- all things that the OS does well and i find no reason to reinvent.

Unless you used a specialized, customized version of netsend, yes it was rate limited -- it by default waits on the clock interval, and warns you if you try to call it in defiance of that. Further it is endlessly calculating the time and calling system functions to get the time.

As a test of max throughput, it is a horrible test. I don't have the motivation to prove it, but I would be surprised if more than 5% of the CPU load actually went towards networking, the rest time calculations and interval tests.

Yea, Im not sure thats a good/accurate bashing. This is a highly optimized tool for a very specific task. OSes have very very wide use cases they have to support.

It's a good and accurate bashing, irrespective of what this tool can do. Even more so once you remember that the networking equivalent of Moore's law tapered off about 10 years ago.

Reply on news.ycombinator.com