The phrase "security through obscurity" actually means more than just the idea that relying on obscurity is a bad idea (which it is). It also points out the problem inherent to security models that aren't holistic. Often these kinds of half-measures give a false sense of security, which can be worse than no security at all. Consider that "security" is not a magical number that you compute from adding up all the various countermeasures you place in front of your endpoint.

Security is more like a hydra. Each head needs to stand on its own and provide real difficulty against attacks, otherwise they'll just be lopped off one by one until the beast is dead. With regard to obscurity, it's like as if one of the heads of the hydra was shriveled and weak. It doesn't even put up a fight. It'll be tossed aside by any attacker that puts any effort in, so it's really not even worth doing. Those naive ssh port scans you mention are the equivalent of kids kicking dirt; nothing a hydra should be worried about.