My concern is that if everyone is in the passive investing boat then we're no longer following the market, we're making the market, and it's a big departure from the philosophical under-pinnings behind the idea of passive investing.
(We started out letting active players make the market by placing good/bad bets and winning/losing. We got a market that was at least trying to find the right price and we did well due to the low expense ratios. Now that we're in the majority, I'm concerned that no-one is trying to find the right price anymore.)
tldr; I have mixed feelings about passive investing over the long term if we're no longer small fish in a big ocean.