I don't know how scientifically valid this is (I hope very) but when a friend told me my USB hub / switcher would be introducing a lot of input lag, I bought a USB to eth adaptor and did a few thousand pings to the router, from direct to mobo and then from via the switcher. Unsurprisingly, there was no measurable latency (I had to use a 3P tool because Windows wouldn't go lower than ms by default).

I am aware that admitting to using Windows in these hallowed halls is a terrible sin, but the anecdote was too relevant to pass up and that's an important detail for anybody looking to repro.