It's my understanding that this works with the complete address, not just the first byte, where it begins processing before the whole frame is received.

On 100Gbit you're talking 0.01ns per bit, so 0.32ns for a full IPv4 address. This compared to, potentially, 15.24ns for a complete IPv4 frame.

But more than that, the encoding used in 10GBASE-T doesn't allow decoding individual bytes. For the usual copper standard, it's 4 pairs with 7 bits per symbol, and the skew between lanes means you can't count on corresponding symbols arriving at the same time. And then there's a reed-solomon block code to correct errors, which has to be decoded all at once. The optical standards use 64b/66b coding, which means you receive blocks of 66 bits and decode 64 out of them (which prevents having too many zeros in a row which can disrupt timing.)

It's nonsense when talking about bytes. As others have pointed out, most switches/routers work with chunks of 16-64 bytes at a time, not singles.