But I for example grew up in a society (Eastern Europe before 1989) where my dad had to wake up early at 4.45 or 5AM in order to go the State-owned grocery store, hoping to catch one bottle of milk for child-me out of a limited supply of 100 or 200 bottles (or something like that). That was a hell of a "zero-sum game" (you wake up early and you're lucky, you get a bottle of milk for your kid out of a limited supply, you don't, your kid is out of milk). As such, it's very difficult for me (and close to impossible) not to view the other people as potential adversaries.
In other words, viewpoints like the one you suggest ("everyone else is not an adversary") most probably come from a position of privilege, i.e. of having grown up in a society which afforded itself the luxury of not having everything as a zero-sum game.