(Allowing it to take 12 billion years to move an inch is rather useless, but that's the same issue with the ephemeralization argument in general: it ignores all other measures of "cost".)
This may seem silly, but microcontroller shrinking is running into a similar problem with quantum noose, which is by far the closer boundary.
The accelerating expantion of the universe also importantly bounds availible time, which makes the specific equation governing how fast optimization can be achieved relevant. For example, if the cost of increasing intelligence is sufficiently exponential past a certain point, a true singularity may never come. And if it's more like busy beaver then exponential then it may be impossible for it to ever get very far indeed.
(Well, as far as we know, anyway. Expansion is still the best fitting theory, but there are observable anomalies. Your argument would still win in some other form, perhaps involving Planck Length or similar.)