Consider the extreme example: Let's say 99.999999% of assets are in index funds and now there are basically only 10 people left in market who are doing active investing. So when these 10 people decides to sell stock of some company, that stock will fall by some minor amount but immediately all index funds would need to react to this minor drop and sell their holdings. But that would cause more market drop and price would take a death spiral route. This basically means that those 10 people now left to do active investment can now control the market. Any of their small movements gets amplified by large gorillas of index funds.
Above is just tip of the iceburg. The secret of the success of index funds has been very simple: They aggregate all the active investors and these average turns out to be better than individual. However when number of active investors reduces, the variance would increase and this means index funds would become less and less optimal over time.