It probably went both ways to some extent. But the Neanderthal population was much smaller, and probably martially weaker, as they were clearly playing a losing evolutionary strategy.