Hacker News Reader

Comment by Animats | Hacker News Reader

I'm surprised that 1-bit works. Trinary (-1, 0, 1) makes sense, but if you only have 0 and 1, the asymmetry ought to be a problem. Or is multiplying an XOR, so 1x1 = 0?

View on HN

It seems like they have learned floating point parameters x and y so that dequantized(bit 0) = x and dequantized(bit 1) = y. Thus there is no built in asymmetry. Or more precisely they learned a zero point and a scale but it's equivalent to this simpler model in the 1 bit case.

It still seems like there would be a problem because either [x, y] looks like [0ish, 1ish] and you can't have negative weights, or [x, y] looks like [-1ish, 1ish] and you can't have "don't care" weights. But if you have some redundancy in your neurons I guess this is acceptable because you can just cancel out the positive contribution from a neuron you don't care about with a negative contribution from a very similar neuron that you also don't care about.

It's more like, addition is an XOR. (In fact, AND as mult and XOR as add are GF(2)). Throw in NOT (or, really, just a constant 1) and you can compute any circuit.

Biologically, inhibitory neurons are every bit as important as excitatory ones, so if you squint just right, XOR looks like a neuron's activation being inhibited by another presynaptic neuron.

There's a scale and bias afterwards, so its not necessarily asymmetric.

Yeah basically. In binary, multiplication is an XNOR operation.

00 = 1

01 = 0

10 = 0

11 = 1

0*0 = 1? I've always hear it as being the output of AND

For balanced binary [-1, 1] yes. -1*-1=1

That makes sense, thanks!

XNOR does not distribute over AND or any other binary operators (try 0 XNOR (0 AND 1)), nor does it have a multiplicative identity, so it's not really multiplication in a way that's useful.

Reply on news.ycombinator.com