Some attackers scan all ports for hosts that are known online through some other port/service/registration. They can also observe your traffic flows at each hop.
Brute force scanning all ports on all IPs might take you a month. But after that you have all these easy targets that people don't patch because they think their server is invisible.
Putting a painting in front of a safe does not make your safe safer. Every thief knows safes are hid behind paintings.