Doesn't infrastructure have users too? Why wouldn't 'user story' mean something to them?

All else equal, smaller or shorter feedback loops are better. And often shortening a feedback loop is worth considerable tradeoffs elsewhere.

It's certainly the case that being able to iterate fast may not be currently possible, but being able to do so, or even just being able to iterate faster should probably always be a priority (even if not the top one).

This all applies even to physical real-world infrastructure. I remember a story about a city somewhere (South America?) where the mayor (?) was able to shorten the feedback loop whereby roads were built or repaired. That seems like a good, even great, thing, even at somewhat considerable cost.

What kind of big infrastructure projects were you thinking of specifically?