Do it now! Modify /etc/ssh/ssh_config.
Do it now! Modify /etc/ssh/sshd_config. Do it now! Modify /etc/ssh/ssh_config.
Do it now! Modify /etc/ssh/sshd_config.Ignoring my opinion of _not_ using a non-standard port, I would update both of the files you listed.
One on the server, to switch to the new port. And one on the client, to modify (or create) a configuration section that (among previous options) now lists a non-default port.
Who in his right mind would want to give the port manually on ever connection attempt? :)
Create the file ~/.ssh/config, then fill it like this:
Host mymachineip
Port 443