Hacker News Reader

Comment by ggm | Hacker News Reader

Turn on Jumbograms. I ran iSCSI for ZFS served out via NFS on distinct cards so I had less contention for the disc fetch against the NFS serve, and it worked "ok" but that was FreeBSD (partly) and 9000 MTU definitely made a difference. Possibly right-sizing the MTU to be bigger than the blocksize is a tuning distinction but 9k jumbo definitely improved things.

Why send 4 packets when one will do? Same volume of data, less switch burden to latch it through.

View on HN

After years in enterprise storage with endless performance testing: there's almost no point. Modern CPUs and NICs barely benefit. In 2005 when we had dual-core CPUs that were constantly buried and NICs with 0 offload, it made a ton of sense - heck we had dedicated iSCSI HBAs (QLA4010 represent!).

That being said, if you've got three servers and two VLANs with no worries about the jumbos ever escaping: I guess? But if you see even a 5% performance increase, I'll be shocked. On the flip side you're one misconfiguration away from endless troubleshooting if those jumbos escape.

Also *jumbo frames.

That's great feedback. If it doesn't help, don't do it.

It'd be great to have an MTU of say 64 KiB or greater.

Although I guess you'd also need a longer than 32-bit CRC to detect all the possible 3 bit errors past 11 kB frame size. A 40-bit CRC would be sufficient, at least up to 188 kB frame size or so.

Some people say for TCP, smaller packets give better acknowledgement pacing. ISCSI is kinda over local, single switch links for me, but general purpose TCP streamed data it may well be "smaller is better" for rate estimates and window management

Nothing prevents you from using a smaller MTU... well "TU" for TCP frames.

Less packets to process would speed things up. Fever headers to process.

Also if your TCP flow bandwidth is counted in tens or hundreds of gigabits per second, there's still going to be plenty of ACKs.

MSS (maximum segment size) is the term at the TCP layer. Each end of a connection can (and usually does) declare its MSS in a TCP option in the first packet it sends.

Advertised MSS, interface MTU, and route MTU can all constrain packet sizing.

Using large-MTU routes for internal destinations can work well.

Right, vaguely aware of that, but been a while. Thanks for correcting!

See perhaps "Best CRC Polynomials":

* https://users.ece.cmu.edu/~koopman/crc/

Thanks. Pretty nice!

If we were redoing Ethernet I wouldn't mind removing the CRC completely. If you want end to end reliability you should do it in the layer above ethernet. If you want link per link packet validation in a network we've already been layering advanced FEC algorithms at the physical layer for high speed Ethernet. The advantage of the latter being it's both optional and replaceable without requiring even more dynamic bits or redundant functionality in the layer 2 packet. Then on MTU make it a 32 bit field instead of a 16 bit field in case anyone wants to make hardware that supports more than 64k in the future.

What amount of packet loss were you experiencing in your setup?

Low enough we had viable mounts before, but the retransmit counts were big. I don't have the host any more, moved to an ix system truenas. Probably I should have looked harder on the provider side.

Reply on news.ycombinator.com