The problem with this explanation is that I should perfectly well be able to divide my jellybeans into 0.5 or -1 groups, though those also make no real sense in this example.
Changing the sign is harder, but let's say your group of jellybeans was actually a loan, and you borrowed from two people... beans / -2