It'll add some minor noise but if you suspect it to be at significant factor you can effectively completely eliminate the noise by picking the lowest values over multiple runs. Something like mtr will report jitter statistics for you.