It also means ISPs are effectively limited to offering the same quality of customer service that the wholesale provider offers, because they need/choose to use the wholesale provider for technician work. That kind of thing can be quite problematic, at least judging by the kinds of complaints you read online.
Depending on the nature of the access network, it might also result in an outdated network seeing no further investment. The UK is only just getting a fibre roll-out going in earnest, and this has come from lots of government encouragement, as well as private new-entrant providers increasingly deciding to just lay their own fibre in. That, plus a reasonably widespread hybrid fibre coax (cable) network offering up to approx gigabit downstream services, providing widespread competition.
I would definitely agree that opening up passive infrastructure (ducts and poles, etc.) would help, as it makes it cheaper for new entrants to rapidly deploy coverage. I think people need to be careful what they wish for though - if the only infrastructure available is outdated copper that can barely manage ADSL2 speeds - where will the investment come from to get to fibre?
If there's already a passive optical network deployed then this is less of an issue for now, but I think it's important to note that, at least in the UK market, the impact of competitive provision has been pivotal to getting fibre rolled out quicker. Without rivals raising investment and digging up roads to lay fibre in large cities, the incumbent "spun out" wholesale provider was calling VDSL "fibre to the cabinet" a "superfast fibre" connection. Once rivals did that, competitive pressures meant FTTP deployment followed.