Basically, there are two classes of people:
1. Those who use port scanners.
2. Those who do not.
If you are being attacked by someone in class 1, then moving your port gives you absolutely no protection. Thus moving your port is only worth anything at all if the percentage of class 2 people is significant.
However, if you also consider the probability a person in each class has of actually compromising your machine, then the security looks less convincing. Yes, it might be true that 95% of people don't bother to use a port scanner, but the most competent hackers are almost certainly going to be in the 5% that do use one.