OK here's how I think it works: from the wikipedia page on two's complement[1], the sign bit is the bit directly to the left of the most significant bit (not the first bit in the 32 bit representation), giving:

  5 => 0101

And the only way I can get this to give -6 is if the bitwise NOT operator takes this entire representation (0101), inverts the bits, then creates a new sign bit and inverts that, giving:

  ~5 => 11010

[1]: https://en.wikipedia.org/wiki/Two's_complement