The only way to visualize it easily is with a physical model. It is all about the angle of the orbits (planes) around the two spheres. You want to minimize 'plane change maneuvers' as they cost a huge amount of fuel (ie they require lots of Delta-V). Think of the earth orbit (the parking orbit before going to the moon) as a big piece of paper around the earth. The available moon orbits are where that piece of paper also touches the sphere of the moon. Now rotate the paper around the earth (the limited number of orbits from florida, between 25 and 90). As the paper rotates around the earth it also rotates around the moon. Some parts of the moon, as some parts of the earth, are not available.

Then add to this the sunlight issues. Apollo wanted to land in very specific light conditions. They wanted nice easy shadows (45) to increase situational awareness during decent and landing. The pilots practiced with the sun being at a particular angle, using surveillance photos taken with the sun at a similar angle. If the sun was too low/high then they might think a rock is bigger/smaller because of its shadow.

If you go into an equatorial orbit, then every orbit of the orbital vehicle gives you a chance to get the lander back.

There is also a stable polar orbit where the orbiter stays lined up between the Earth and the Moon. Since the surface is tidally locked you again get the lander back easily. This orbit takes more energy to get into and out of for the orbital vehicle. But gives you mid latitudes with the Earth high in the sky.

What is hard is getting to mid latitudes on the side of the Moon. Now what happens is that the object on the surface turns away from the orbital vehicle making it hard to line up everything for the return trip. The lander therefore has to be much more capable to land and return at will, rather than waiting weeks for things to line up right.

The problem was not landing. The problem was getting back into space, to rendezvous with the command module.

The reason this is difficult is because orbital plane changes (changes in your orbit's inclination) are expensive, and because astronauts can't sit on the lunar surface for weeks at a time.

Launching to rendezvous with a space space station (Or the command module) is, if you want to avoid very expensive plane changes, requires waiting for it to be a few minutes from passing directly overhead. Then, you launch into the same orbital plane as the space station, and, in the process of accelerating to orbital speeds, the space station catches up to you, and you dock to it.

If the command module is in an equatorial orbit, and you land on the equator, then rendezvous is as easy as taking off from the lunar surface, a few minutes before it is scheduled to pass overhead. This happens every ~2 hours.

If the comand module is in a polar orbit, and you land on the poles, then the same thing happens. The orbiter will be overhead every ~2 hours, so, again, rendezvous is very easy.

But if you land in the mid-latitudes, you suddenly have a problem.

The moon rotates. And it rotates slowly.

Imagine if the command module is in a polar orbit, and you land in the mid-latitudes. You spend two days driving golf carts on the moon. You want to get back to space, to meet up with the command module.

Except that the moon has rotated in this time. But the polar orbit hasn't. Instead of passing overhead, the command module is now one-tenth of the way around the moon from you. You can't launch into its orbital plane anymore.

You now have two options.

1. You can wait for a total of 13.5 days, when due to the moon's rotation, you will, again, be within the command module's orbital plane.

2. You can launch into an orbital plane that is ~30 degrees inclined to that of the command module. And then spend a tonne and a half of rocket fuel, to change your orbital inclination.

Both of these options, for obvious reasons, suck.

Now, option #1 gets a bit easier, if the command module is in a non-polar orbit - the launch window, instead of being every 13.5 days, may be every few days, or once a day.

However, if you need to, for whatever reason, perform an emergency launch, or if you miss the launch window, you may be screwed.

If the Apollo mission did not involve a command module (If the lunar lander was the craft that would come back to Earth), then all of this becomes irrelevant. You could then land at whatever latitude, and return to Earth, via a wide range of possible trajectories. (Except that, as another poster pointed out, you miss out on a part of, or all of the free-return trajectory boost, that you get from exiting the Moon's orbit in the opposite direction of its travel - which is ~1,000 m/s of delta-v saved.)