This lag is introduced by vim, to wait for the next keys and see if you are actually inserting them or using mapped key sequence.
It is configurable in vim, but then either you have a lag and you are able to use your key sequence, or you don't and <leader> sequences cannot be used.
So maybe I have something configured that sets it wrong, but I do not see how to reconcile both things.