Body armor is heavy and obvious, especially in warm climates. I don't think it would start an arms race. You would only wear it when you are attacking or expecting an attack. Not to mention it only protects your chest/back.
Body armor comes in a wide variety of sizes and weights many of which are not heavy and obvious. You don't need a full getup with plates to stop handguns.