But the yardstick argument assumes uniform expansion. Small scale structure like our yardstick is held together by electromagnetic forces which counteracts the expansive force. That's my understanding anyway.

If we create a light year long line of yardsticks then we would expect the line to expand faster than the yardsticks that make it up. Locally this would look like growing gaps between the yardsticks. we never used the fact that the line was a light year long. So let's just put two yardsticks next to each other in a vacuum. They drift apart and eventually so far they are not part of each others' observable universes.