There are a variety of factors that add up to many ms of lag. USB and HDMI are a few that come to mind. USB is incredibly complex when compared to e.g. ps/2. If you are building an embedded board like rasberry pi and you want to add ps/2, you just add a couple pins. If you want to add usb, you buy an extra chip.