Hacker News Reader

Comment by agapron1 | Hacker News Reader

An 8 bit adder is too small and allows such 'hack'. He should try training a 32 or 64 bit adder, decrease the weights accuracy to bfloat16, introduce dropout regularization (or other kind of noise) to get more 'interesting' results.

Addendum: another interesting variation to try is a small transformer network, and feeding the bits sequentially as symbols. This kind of architecture could compute bignum-sized integers.

View on HN

Its converted the input into its native language, in a way

A 32-bit adder is just 4 8-bit adders with carry connected. I don't see why it'd be significantly more difficult.

Yes, but the adder that the NN came up with has no carry.

Carry is just the 9th bit?

Not really, to compose adders you need specific lines for carry.

It's better to say the NN "converged to" something than "came up with" something.

Why?

There is no intelligence or reasoning involved in stochastic gradient descent

It's not harder computationally but finding that solution with SGD might be hard. I'd be really interested in the results.

In my EE class that discussed adders, we learned this wasn't exactly true. A lot of adders compute the carry bit separately because it takes time to propagate if you actually capture it as output.

For a naive adder, sure. Actual adder circuits use carry lookahead. The basic idea is to “parallelize” carry computation as much as possible. As with anything else, there is a cost: area and power.

Edit: This wiki page has a good explanation of the concept: https://en.wikipedia.org/wiki/Carry-lookahead_adder

Yeah, the current solution is similar to overfitting, this wont generalize to harder math where the operation doesn't correspond to the activation function of the network.

Is this true? If it can add, it can probably subtract? If it can add and subtract it may be able to multiply (repeated addition) and divide? If it can multiply it can do exponents?

I don’t know, but cannot jump to your conclusion without much more domain knowledge.

> If it can add and subtract it may be able to multiply (repeated addition) and divide? If it can multiply it can do exponents?

It was just a simple feed-forward network. It can't do arbitrary amounts of repeated addition (nor repeat any other operation arbitrarily often).

None of these require arbitrary amounts of repeated addition though. E.g. multiplying two 8 bit numbers requires at most 7 additions.

Are you doing the equivalent of repeated squaring here?

Otherwise, you'd need up to 255 (or so) additions to multiply two 8 bit numbers, I think?

An 8x8 bit multiplication only requires 7 additions, either in parallel or sequentially. Remember long-form multiplication? [1] It's the same principle. Of course, high-speed digital multiplication circuits use a much more optimized, much more complex implementation.

[1] https://en.wikipedia.org/wiki/Multiplication_algorithm#Examp...

OK. It sounded to me like the comment I originally replied to (https://news.ycombinator.com/item?id=34400296) suggested to implement multiplication as naive repeated addition.

Yeah I did not mean a loop, repeated in the network.

Repeated additions may be hard, but computing a*b as exp(log(a)+log(b)) should be learnable.

A sequential one-bit adder, like in pocket calculators or earliest computers with delay-line memory, could likely be invented in the process.

Reply on news.ycombinator.com