jQuery adds padding for browsers that don't support certain features or have certain quirks (yes, IE, we're mainly looking at you). This is overhead that will slow things down.
That being said, if you can afford to develop for modern browsers only, then by all means, use plain JS.