I think the biggest fear is lack of accountability.

We have already seen this causing real problems for real people. Please see Jeremy Howard's lecture on ethics in AI for some real world examples.