Hacker News Reader

Comment by geerlingguy | Hacker News Reader

It looks like the problem is `ksoftirqd` gets pegged at 100% and the system just queues up packets, slowing everything down. See: https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...

View on HN

I would suggest you go ahead and try jumbo frames[0] as that will significantly decrease the CPU load and overhead.

I would also suggest using taskset[1] on each iperf server process to bind them each to a different cpu core.

Finally, I would suggest UDP on iperf and let the sending Pi's just completely saturate the link.

If you do all that, I think you have a good chance at achieving 3.5Gbps over just the Intel card.

0: https://blah.cloud/hardware/test-jumbo-frames-working/

1: https://linux.die.net/man/1/taskset

This is common even on x86 systems.

You have to set the irq affinity to utilize the available CPU cores.

There is a script included with the source you used to compile drivers called "set_irq_affinity"

Ex (Sets IRQ Affinity for all available cores) :

[path-to-i40epackage]/scripts/set_irq_affinity -x all ethX

So like https://pastebin.com/2Z4UECPq ? — this didn't make a difference in the overall performance :(

Looks like the script needs to be adjusted to function on the Pi.

I wish I had the cycles and the kit on hand to play with this!

So, this is sorta indicative of a RSS problem, but on the rpi it could be caused by other things. Check /proc/interrupts to assure you have balanced MSI's, although that itself could be a problem too.

edit: run `perf top` to see if that gives you a better idea.

Results:

    15.96%  [kernel]                      [k] _raw_spin_unlock_irqrestore
    12.81%  [kernel]                      [k] mmiocpy
     6.26%  [kernel]                      [k] __copy_to_user_memcpy
     6.02%  [kernel]                      [k] __local_bh_enable_ip
     5.13%  [igb]                         [k] igb_poll

When it hit full blast, I started getting "Events are being lost, check IO/CPU overload!"

Another idea will be to increase interrupt coalescing via ethtool -c/C

Reply on news.ycombinator.com