That is the thing, if we are talking about my personal linux server I am good with key auth + fail2ban and ssh on high port to have less spam in logs.

For work related stuff if I have to deal with multiple people accessing multiple servers then it is different story. Strictly forbidding ssh to have people connect from one IP is a lot more control. Then also key auth but I don't have to setup some google auth.