You want your brightest in society to do things that further society/humanity as a whole; playing the stock-market game or optimizing models to further consumerism (which is what most ML in the industry boils down to, let's be honest) isn't exactly doing that.
Everyone's absolutely free to work on whatever they want, that's for sure. But from a society point of view, we should ask ourselves if our system sets the right incentives.
It definitely is a social issue, the main reason people make that switch is money and money is a social issue. Why would you make less money doing physics research than helping advertisers better manipulate people? There is no reason, currently society rewards the later more than the former, but in another society it could be the other way around.
So the problem isn't if you as an individual choose to switch field since you'd like that field better, but all those who switched since that field pays better. That is the social issue.