It's probably more than 'a little' latency, try:

openssl speed rsa2048 rsa4096

on your server for what is probably going to be a rude awakening. For a small, low-traffic site you might be able to manage but as traffic increases you are probably going to regret choosing 4096 bit certs unless you absolutely need them for a specific reason.