Externalities usually become obvious after the fact. The problems we aim to solve through our inventions are multi-dimensional topologies. Like tree, at any given point we can only see portions of it, leaving other perspectives unaccounted. Trimming leaves or branches would reveal deeper previously unseen parts.

The human brain and our tools thus far are more adapted to solving linear problems, so the higher the dimensionality and non-linear the relationships, the less we're able to produce determinist results.

I personally believe problems of this kind we cannot arrive at perfect solutions, only optimal ones. By optimal, I mean making trade-offs on certain variables so as to minimize total cost. The weights are also variables depending on trends of the epoch (environmental factors not such a big deal a decade ago, big deal now --> need to update our equation)

The other factor comes as a form of bureaucracy. It takes time to push the change through the chain of politics, re-aligning incentives, carving new markets and distribution channels, changing market sentiments, etc.