True - buildings and exposed rock probably stay still, leaf litter and trees definitely don’t. You’d need to find enough of a stable reference to run your correlation - probably more useful in an urban area.
On a millimeter scale I expect weather/seasons will cause movement of the ground as the soil swells with water from rain, or water freezing.
Taller buildings can flex up to a few feet in each direction actually!
I believe tectonic plates generally move about 15mm a year - which is pretty fast when you think about it!
> Plate motions range from 10 to 40 mm/year at the Mid-Atlantic Ridge (about as fast as fingernails grow), to about 160 mm/year for the Nazca Plate (about as fast as hair grows)