Basically, there are two classes of people:
1. Those who use port scanners.
2. Those who do not.
If you are being attacked by someone in class 1, then moving your port gives you absolutely no protection. Thus moving your port is only worth anything at all if the percentage of class 2 people is significant.
However, if you also consider the probability a person in each class has of actually compromising your machine, then the security looks less convincing. Yes, it might be true that 95% of people don't bother to use a port scanner, but the most competent hackers are almost certainly going to be in the 5% that do use one.
ie. 1. is targeting you specifically, 2. is bot targeting everyone
when presented with two options, thinking of those options as 50:50 is natural, but it's really more like 0.0000001:99.9999999
As a consequence, the chance of one of those 0.999999 bots invading your computer is zero, the chance of one of the 0.000001 doing the same is non-zero.
http://nmap.org/book/idlescan.html but IIRC there are more ways than this to do it.