I can foresee a future where this (in practice) eliminates choice/individuality because people will allow their responses to be dictated by an AI for "best results."

Then whoever controls the AI controls society.

Tag onto that how AI/ML often reinforces systemic problems, and one can foresee how it would further entrench the status quo.