I'm pretty sure that the different semantics for negative signed ints (sign extension) are there to make sure it works as division. It's right-shifting for regular bit manipulation purposes that's unsafe with signed.
For the curious who don't want to duplicate my look-up - right shift is by definition division by two except negative signed numbers, for which it's implementation defined. GCC on every architecture I've checked uses arithmetic shift for signed, probably because that extends the base definition to negatives.