Until motherboards include SFP ports it's probably not worth the effort at all in home setting; external adaptors like the one presented here are unreliable and add several ms of latency.
A micro-ATX motherboard with on-board 2xSFP28 (Intel E810):
* https://download-2.msi.com/archive/mnu_exe/server/D3052-data...
* https://www.techradar.com/pro/this-amd-motherboard-has-a-uni...
Where did you get "several ms of latency" figure from? I have not measured external card, but may be I should do it... Because cards themselves have latency in range of microseconds, not millis.
there are a lot of usb options that matter, and tp-link ships lots of realtek chipsets that require very special driver incantations that a lot of the linux drivers simply don't replicate.
two+ layers of bad options will surely add 4ms quick.
USB itself can have a lot of issues anywhere in the chain. I have a Thunderbolt dock where half of the USB ports adds latency and reduced throughput just because the USB chipset that powers them is terrible (it has two separate USB chipsets from different brands). Switch to a different port on the exact same dock and it's fine.