Unless your whole chain is 4096 bit - which is unlikely - a 4096 bit RSA key just adds a little latency.
openssl speed rsa2048 rsa4096
on your server for what is probably going to be a rude awakening. For a small, low-traffic site you might be able to manage but as traffic increases you are probably going to regret choosing 4096 bit certs unless you absolutely need them for a specific reason.