> Won’t the light hit "different atoms" on the mirror, so to speak, thus changing the distance travelled by much more than fractions of the size of a single atom?

Yes, it will, but that is already represented in the interference pattern.

They're not measuring the absolute distance to the mirror. If they were, you are right about how the precision would be limited.

Instead they're using the interference pattern to measure a change in the distance measurement over time. So even though the distance is somewhat of an average over many atoms, as long as it is the same mirror, it will be the same average at the same distance.

Because the interference pattern represents photons interfering with each other, its precision is limited by the size of photons--which are much smaller than atoms.