Even if it didn't, it lowers the number of log entries, failed attempts, and number of connections by several orders of magnitude. I've never regretted doing it though I pick a far more obscure port than this. None of my clients have had issues with me doing this either and sshagent makes it transparent to me.
It's the equivalent of moving your main door to the back of your house. Far fewer people will knock, even though burglars can still get in.