I think the main criticism is that centralizing the “legitimate” use of force to, as you say, “keep everyone in line”, is that you get all of the downsides of a violence-based society, and none of the upsides of freedom: such societies still enslave, torture, steal, violate privacy, and mass murder.

Indeed, we’re witnessing an unprecedented wealth transfer right now in the US as a result of such centralization of authority.

I’ve never met an anarchist that believes that by abolishing the state that society will become entirely nonviolent. It’s just a sort of standard “who will build the roads?” straw man: Policing doesn’t need police, and social contracts don’t need a statehouse.