No, not at all.

What you do notice is using ciphers that are apt for the machine. If both ends are modern x86, for example, then you'd want to use aes256-gcm@openssh.com or aes128-gcm@openssh.com for best performance. Otherwise stick to chacha20-poly1305@openssh.com, which is the default of recent versions.