On the flip side, that means there's nothing to stop the pole owner from delaying, as (if I understand correctly) Google argued was happening here. Sure, you could try to fine uncooperative owners, but the delay will kill small ISPs who can't afford to wait; the potential fine would have to massive to deter such behavior, and between lobbying and general antipathy toward regulation, that size of fine would likely never occur.
I don't disagree though that there should be some form of middle ground (which it seems Pai was trying to find) -- I'm just unsure what it would be.