I think op is saying, at the cost of an undetectable fraction of a second rounding error when you SSH somewhere, so what?
But that argument shouldn't be applied for all use cases, such as personal key management, or keys that are used for days/weeks/months at a time.
I tried switching from a 2048-bit to a 4096-bit key to control a modestly sized VPS (~4GB RAM, 2CPU) and scp file transfer speeds plummeted.
I have a symmetric 1GB FTTH connection and I'm used to everything being pretty quick. Using a longer key was like a return to dial-up speed. If you don't plan to transfer large files or directories you can safely ignore it, but I bet your patience will run out pretty quickly if you do.
(Unless you are saying you somehow convinced SSH to use a symmetric cipher with a 2-4k key size...?)