Hacker News Reader

Its a single lane pcie gen2 interface. The max theoretical is 500MB/sec. So you can't ever touch 10G with it. In reality getting 75% of theoretical on PCIe tends to be a rough upper limit on most PCIe interfaces, so the 3Gbit your seeing is pretty close to what one would expect.

edit: Oh its 3Gbit across 5 interfaces, one of which isn't PCIe, so the PCIe side is probably only running at about 50%. It might be interesting to see if the CPUs are pegged (or just one of them). Even so, PCIe on the rpi isn't coherent so that is going to slow things down too.

It looks like the problem is `ksoftirqd` gets pegged at 100% and the system just queues up packets, slowing everything down. See: https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...

I would suggest you go ahead and try jumbo frames[0] as that will significantly decrease the CPU load and overhead.

I would also suggest using taskset[1] on each iperf server process to bind them each to a different cpu core.

Finally, I would suggest UDP on iperf and let the sending Pi's just completely saturate the link.

If you do all that, I think you have a good chance at achieving 3.5Gbps over just the Intel card.

0: https://blah.cloud/hardware/test-jumbo-frames-working/

1: https://linux.die.net/man/1/taskset

This is common even on x86 systems.

You have to set the irq affinity to utilize the available CPU cores.

There is a script included with the source you used to compile drivers called "set_irq_affinity"

Ex (Sets IRQ Affinity for all available cores) :

[path-to-i40epackage]/scripts/set_irq_affinity -x all ethX

So like https://pastebin.com/2Z4UECPq ? — this didn't make a difference in the overall performance :(

Looks like the script needs to be adjusted to function on the Pi.

I wish I had the cycles and the kit on hand to play with this!

So, this is sorta indicative of a RSS problem, but on the rpi it could be caused by other things. Check /proc/interrupts to assure you have balanced MSI's, although that itself could be a problem too.

edit: run `perf top` to see if that gives you a better idea.

Results:

    15.96%  [kernel]                      [k] _raw_spin_unlock_irqrestore
    12.81%  [kernel]                      [k] mmiocpy
     6.26%  [kernel]                      [k] __copy_to_user_memcpy
     6.02%  [kernel]                      [k] __local_bh_enable_ip
     5.13%  [igb]                         [k] igb_poll

When it hit full blast, I started getting "Events are being lost, check IO/CPU overload!"

Another idea will be to increase interrupt coalescing via ethtool -c/C

>It might be interesting to see if the CPUs are pegged (or just one of them).

This is very likely the answer. I see a lot of people who think of the Pi as some kind of workhorse and are trying to use it for things that it simply can't do. The Pi is a great little piece of hardware, but it's not really made for this kind of thing. I'd never think about using a Raspberry Pi if I had to think about "saturating a NIC".

Well it can saturate up to two, and almost three, gigabit NICs now. So not too shabby.

But I like to know the limits so I can plan out a project and know whether I'm safe using a Pi, or a 3-5x more expensive board or small PC :)

>Sorry about the slightly-clickbaity title.

Well yes because 5Gbps Ethernet is actually a thing ( NBase-T or 5GBASE-T). So 1Gbps x 5 would be more accurate.

Cant wait to see results on 10GbE though :)

P.S I really wish 5Gbps Ethernet is more common.

My ATT router made by Nokia has one 5gbe and the fiber plugs in directly with SFP!

True true... though in my work trying to get a flexible 10 GbE network set up in my house, I've found that the support for 2.5 and 5 GbE are iffy at best on many devices :(

The "best" I've found so far (and gives you options to go 2.5GbE/5GbE

1: https://www.amazon.com/UGREEN-Ethernet-Thunderbolt-Converter... (USB-C)

2: https://www.amazon.com/2-5GBase-T-Ethernet-Controller-Standa... (Don't get the knock-off version of this, the brackets aren't the right sizes.) (PCIe)

The expensive ones I'm waiting to arrive:

3: Either a second hand Intel X520-DA1 card or the "refurb" from AliExpress

and https://mikrotik.com/product/crs305_1g_4s_in with RJ10 SFP+ modules. Then cry at how much you just spent.

X520-DA1 and most older stuff doesn’t support 2.5Gbps or 5Gbps (only 1 and 10).

Oops. They haven't arrived yet. Good thing they weren't too expensive. What would be my best bet for supporting all of those speeds?

Awesome work. Been watching your videos on these (the video card one was especially interesting).

At what point are you saturating the poor little ARM CPU (or its tiny PCIe interface)?

Heh, I know that ~3 Gbps is the maximum you can get through the PCIe interface (x1, PCI 2.0), so that is expected. But I was hoping the internal ethernet interface was separate and could add one 1 Gbps more... the CPU didn't seem to be maxed out and was also not overheating at the time (especially not with my 12" fan blasting on it).

with some tuning you should be able to saturate the PCIe 1x slot.

Excellent reading on this available here :

http://www.intel.com/content/dam/doc/application-note/82575-...

and here :

https://blog.cloudflare.com/how-to-achieve-low-latency/

Edit : with the inbound 10Gb card referenced

Was all this TCP? You might try UDP as well, in case you're hitting a bottleneck in the tcp stack.

I assume you saw the video with Plunkett the RPF put out ( https://youtu.be/yiHgmNBOzkc specifically interesting at 10:45 ) - He mentioned he was testing 10GbE fibre and reached 3.2gbit. Now he goes into absolutely zero detail on that, but I find it interesting you've both hit the same ceiling.

(He also mentioned 390MB/sec write speed to nvme, which is suspiciously close to the same ceiling)

Yeah, I think the PCIe link hits a ceiling around there.

Note that combining the internal interface with the 4 NIC interfaces, and overclocking to 2.147 GHz got it up to 3.4 total Gbps. So the IRQ interrupts are the main bottleneck when it comes to total network packet throughout.

Since you're also from the midwest, I'll put it in terms you'll understand: :-)

> I think the PCIe link hits a ceiling around there.

You're trying to shove 10 gallons of shit into a 5-gallon bucket!

--

I'm not sure how high you can set the MTU on those Pi's (the Intels should handle 9000) but I'd set them as high as they'll go, if I were you. An MTU of 9000 basically means ~1/6th the interrupts.

Jeff,

First off, thank you for doing this kind of 'r&d', it is really exciting to see what the Pi is capable of after less than a decade.

Would you be interested in someone testing a SAS PCI card? I'm going to pick up one of these as soon as they're not backordered...

Do you think an SFP+ nic would work? It would be cool to try out fiber.

There are no SFP option on 5gbps NICs as i understand as per standard

You might be hitting the limits of the RAM. I think LPDDR3 maxes out at ~4.2Gbps, and running other bus masters like the HDMI and OS itself would be cutting into that.

32-bit LPDDR4-3200 should give 12.8 Gbytes/s which is 102 Gbits/s.

You can't just multiply width*frequency for DRAM these days, as much as I wish we still lived in the days of ubiquitous SRAM.

The chip in some of the 2GB RPI4s is rated for only 3.7Gbps.

https://www.samsung.com/semiconductor/dram/lpddr4/K4F6E304HB...

No, that chip is rated for 3.7 Gbps per pin and it's 32 bits wide. Even at ~60% efficiency you're an order of magnitude off.

Real world tests are seeing around 3 to 4 Gbps of memory bandwidth.

https://medium.com/@ghalfacree/benchmarking-the-raspberry-pi...

LPDDR cannot sustain anywhere near the max speed of the interface. It's more of a hope that you can burst something out and go to sleep rather than trying to maintain that speed. In a lot of ways DRAM hasn't gotten faster in decades when you look at how latency clocks nearly always increase at the same rate of interface speed increases. And LPDDR is the niche where that shines the most, because it doesn't have oodles of dies to interleave to hide that issue.

Innumeracy strikes again. It's actually 4-5 Gbytes/s [1] plus whatever bandwidth the video scanout is stealing (~400 Mbytes/s?). That's only ~40% efficient which is simultaneously terrible and pretty much what you'd expect from Broadcom. However 4 Gbytes/s is 32 Gbits/s which leaves plenty of headroom to do 5 Gbits/s of network I/O.

[1] https://www.raspberrypi.org/forums/viewtopic.php?t=271121

Those numbers look way off, maybe they mixed up the units? Should be a few GBps at least.

Bits aren't bytes.

The y axis is labeled "megabits per second".

The y axis is wrong.

Is there a way to see if you are hitting memory bandwidth issues in Linux?

Not in a holistic way AFAIK, and for sure not rigged up to the Raspbian kernel (since all of that lives on the videocore side), but I bet Broadcom or the RPi foundation has access to some undocumented perf counters on the DRAM controller that could illuminate this if they were the ones debugging it.

Instead of lying and then apologizing once you get what you want, it would be better to just not lie in the first place.

Technically it's not a lie—there are 5x1 Gbps of interfaces here. But I wanted to acknowledge that I used a technicality to get the title how I wanted it, because if I didn't do that, a lot of people wouldn't read it, and then we wouldn't get to have this enlightening discussion ;)

You could hook up a 100gbs card, but that wouldn't make it 100gbs ethernet on a raspberry pi.

It would, but it wouldn't be able to push 100gbs. It's not lying

I'm just looking around for this mystery 100 Gbps card. Think it'll work with Cat9?

Did you think this was made up for some reason? 100Gb cards are not new. QSFP28 was created in 2014.

https://www.broadcom.com/products/ethernet-connectivity/netw...

They don't use copper, they use fiber. It wouldn't be a mystery if you searched for '100gbs pci ethernet'.