To make things more complicated: the size of your desk isn't fixed because the universe is expanding. So the units you are using are just a convention to relate the size of an object to the size of other objects.
So long as the fine structure constant remains the same, your desk (and ruler) won't change size. Rather they will heat slightly as the expanding universe stretches the chemical bonds and those same bonds spring back to average rest length.
/nitpick
Expressing the size of the desk in a coordinate system specifically chosen such that the size doesn't change is much more than a convention. Conventions are arbitrary, like whether to drive on the left or right side of a two-way road.
The effects of air eroding the desk are much more significant than the expansion of the universe. And both are dwarfed by the fact that the texture of the surface keeps the exact length and width from being well-defined.
Also, I tend to move around while my desk stays still, so from my desks point of view I am not aging as fast.