I can get it down to 1ms, so there's probably something wrong with my setup. (I just look for how the cursor moves; on the delayed one, I can discern the presence of the cursor on the next line for a split second.) Maybe monitor refresh rate, or the minimal resolution sleep can handle?
I'm using Alacritty, which is supposed to be really fast and everything!