Hacker News Reader

Comment by greenbit | Hacker News Reader

That's awesome - the network's solution is essentially to convert the input to analog, perform the actual addition in analog, and then convert that back to digital. And the first two parts of that all happened in the input weights, no less.

View on HN

An 8 bit adder is too small and allows such 'hack'. He should try training a 32 or 64 bit adder, decrease the weights accuracy to bfloat16, introduce dropout regularization (or other kind of noise) to get more 'interesting' results.

Addendum: another interesting variation to try is a small transformer network, and feeding the bits sequentially as symbols. This kind of architecture could compute bignum-sized integers.

Yeah, the current solution is similar to overfitting, this wont generalize to harder math where the operation doesn't correspond to the activation function of the network.

Is this true? If it can add, it can probably subtract? If it can add and subtract it may be able to multiply (repeated addition) and divide? If it can multiply it can do exponents?

I don’t know, but cannot jump to your conclusion without much more domain knowledge.

> If it can add and subtract it may be able to multiply (repeated addition) and divide? If it can multiply it can do exponents?

It was just a simple feed-forward network. It can't do arbitrary amounts of repeated addition (nor repeat any other operation arbitrarily often).

None of these require arbitrary amounts of repeated addition though. E.g. multiplying two 8 bit numbers requires at most 7 additions.

Are you doing the equivalent of repeated squaring here?

Otherwise, you'd need up to 255 (or so) additions to multiply two 8 bit numbers, I think?

An 8x8 bit multiplication only requires 7 additions, either in parallel or sequentially. Remember long-form multiplication? [1] It's the same principle. Of course, high-speed digital multiplication circuits use a much more optimized, much more complex implementation.

[1] https://en.wikipedia.org/wiki/Multiplication_algorithm#Examp...

OK. It sounded to me like the comment I originally replied to (https://news.ycombinator.com/item?id=34400296) suggested to implement multiplication as naive repeated addition.

Yeah I did not mean a loop, repeated in the network.

Repeated additions may be hard, but computing a*b as exp(log(a)+log(b)) should be learnable.

A 32-bit adder is just 4 8-bit adders with carry connected. I don't see why it'd be significantly more difficult.

Yes, but the adder that the NN came up with has no carry.

Carry is just the 9th bit?

Not really, to compose adders you need specific lines for carry.

It's better to say the NN "converged to" something than "came up with" something.

Why?

There is no intelligence or reasoning involved in stochastic gradient descent

It's not harder computationally but finding that solution with SGD might be hard. I'd be really interested in the results.

In my EE class that discussed adders, we learned this wasn't exactly true. A lot of adders compute the carry bit separately because it takes time to propagate if you actually capture it as output.

For a naive adder, sure. Actual adder circuits use carry lookahead. The basic idea is to “parallelize” carry computation as much as possible. As with anything else, there is a cost: area and power.

Edit: This wiki page has a good explanation of the concept: https://en.wikipedia.org/wiki/Carry-lookahead_adder

Its converted the input into its native language, in a way

A sequential one-bit adder, like in pocket calculators or earliest computers with delay-line memory, could likely be invented in the process.

Sometimes when doing arithmetic with sed I use the unary representation, but usually it's better to use lookup tables: https://en.wikipedia.org/wiki/IBM_1620#Transferred_to_San_Jo...

If you want to have some fun, implement code to do arbitrary precision arithmetic the way the CADET did it. It's remarkably simple and reliable.

So that's how savants do it ...

There is one savant that can do crazy math but is otherwise normal. Afgter an epileptic fit.

His system is base 10000, he can add any two numbers below 5000 and come up with a single digit response in one loop. Then covert to base10 for the rest of us. Each digit in his base 10000 system has a different visual representation, like we have 0-9.

It's integer accurate so I don't think it uses sine wave approximations. Guy can remember pi for 24 hours.

I don't think the boffins or he himself got close to understanding what is going on in his head.

I was interested in reading more about this, but the only google result is your comment. Could you remember a name or any other details that might help find the story?

Does this relate to the addition theorem of Fourier Transforms? The weights of his NN looked not only looked like sine waves but were crudely orthogonal.

Maybe in the future we will think of the word "digital" as archaic and "analog" will be the moniker for advanced tech.

Can someone please explain, I don't get here "converted the input to analog", converted how, where?

Reply on news.ycombinator.com