RTL latency doesn't seem that bad on chips that are built for the purpose, like RME's. Just the off the shelf stuff might be less than ideal.

If anything I think a problem is there's just overall too much slop in the whole chain, including the operating system, motherboard, whatever else. It's a little ridiculous that a $50 guitar pedal can easily get better RTL numbers than a $1500 computer system.