Well, frankly I don't think it's a disadvantage. It's difficult to move out from the equilibrium of countries mistrusting each other, but I think what you describe is actually an utopia. If we manage to destroy all tariffs and make countries lose parts of their production abilities, we'd make war unthinkable, which is a net good. The US would never start an all-out war because it would starve, and whomever specialized in the agricultural products supplanting the US productive capability would never start a war because they cannot build weapons.
Of course there are huge problems with this -- what happens if whomever makes your food goes tits up, involuntarily (natural disasters, coups, ...) or voluntarily (maliciously attacking)? Who guarantees the system? How? Etc... --, if there weren't I wouldn't call it utopia. But I think raising the costs of war beyond the unreasonable is a net gain.
Note: I'm fully aware that this is an extremist and probably next to impossible position, no one has to remind me that. I just contest the fact that the objection "but then we could not go around and kill people" isn't a particularly good counter-argument for no trade barriers.