Hacker News Reader

Comment by dagss | Hacker News Reader

Impressive. Am I right that what is really going on here is that the network is implementing the "+" operator by actually having the CPU of the host carrying out the "+" when executing the network?

I.e., the network converts the binary input and output to floating point, and then it is the CPU of the host the network is running on that really does the addition in floating point.

So usually one does a bunch of FLOPs to get "emerging" behaviour that isn't doing arithmetic. But in this case, instead the network does a bunch of transforms in and out so that in a critical point, the addition executed in the network runner is used for exactly its original purpose: Addition.

And I guess saying it is "analog" is a good analogy for this..

View on HN

I'd say it's more that the network "discovered" that addition is something its own structure does "naturally", and reduced the problem into decoding binary input, and encoding binary output. In particular, I don't think it's specifically about the CPU having an addition operator.

My intuition is as follows: if I were to train this network with pencil, paper and a slide rule, I'd expect the same result. Addition (or maybe rather integration) is embedded in the abstract structure of a neural network as a computation artifact.

Sure, specifics of the substrate may "leak through" - e.g. in the pen-and-paper case, were I to round everything to first decimal space, or in the computer case, was the network implemented with 4-bit floats, I'd expect it not to converge because of loss of precision range (or maybe figure out the logic gate solution). But if the substrate can execute the mathematical model of a neural network to sufficient precision, I'd expect the same result to occur regardless of whether the network is run on paper, on a CPU, an fluid-based analog computer, or a beam of light and a clever arrangement of semi-transparent plastic plates.

You know how the basic node does a weighted sum of inputs, and feeds the result to a non linear function for output? This network exploited the addition operation implicit in that first part, the weighted sum.

Fascinating! Is it really exploitation? It seems to me that the node should optimize to the minimal inputs (weights) in the happy path. I.e. it's optimal for all non-operand inputs to be weighted to zero.

I'm curious how reproducible this is. I'm also curious how networks form abstractions and if we can verify those abstractions in isolation (or have other networks verify them, like this) then the building blocks become far less opaque.

That's an accurate enough description of what's going on here, yeah, but I'm very curious if this could be implemented in hardware. What we've got is an inexact not-quite-binary adder, but one that's potentially smaller and faster than the regular binary ones.

It can be implemented in hardware but the implementation would be more complex than a digital adder based on logical gates.

Analog addition is actually really easy if you can tolerate noise and heat and can convert to/from representing the signal as an analog current. Using Kirchhoff's Current Law, addition of N currents from current sources is achieved by joining those N wires to a shared exit wire which contains the summed current.

Addition is easy, DAC is relatively easy, but ADC is not.

Why it works here is that the "analog" representation is not influenced by noise, because it's simulated on digital hardware.

On the other hand, why we use digital hardware precisely because it's robust against the noise present in the hardware analog circuits.

So in other words, this is analog computation on a digital hardware on analog substrate - the analog-to-digital step eliminates the noise of our physical reality, and the subsequent digital-to-analog reintroduces a certain flexibility of design thinking.

I wonder, is there ever a case analog-on-digital is better to work with as an abstraction layer, or is it always easier to work with digital signals directly?

Yes, at least I think so. That's why we so often try to approximate analog with psuedo-continuous data patterns like samples and floats. Even going beyond electronic computing, we invented calculus due to similar limitations with discrete data in mathematics.

Of course, these are all just approximations of analog. Much like emulation, there are limitations, penalties, and inaccuracies that will inevitably kneecap applications when compared to a native implementation running on bare metal (though, it seems that humans lack a proper math coprocessor, so calculus could be considered no more "native" than traditional algebra).

We do sometimes see specialized hardware emerge (DSPs in the case of audio signal processing, FPUs in the case of floats, NPUs/TPUs in the case of neural networks), but these are almost always highly optimized for operations specific to analog-like data patterns (e.g.: fourier transforms) rather than true analog operations. This is probably because scalable/reliable/fast analog memory remains an unsolved problem (semiconductors are simply too useful).

Sorry but those signals aren't analog. Floating point numbers are still digital.

Right, but there exists software that would be robust to noise, e.g. ...neural networks. I'm curious if not-quiet-perfect adders might be beneficial in some cases.

Quad-level flash memory cells (https://en.wikipedia.org/wiki/Multi-level_cell#Quad-level_ce...) can store 4 bits for months.

Because of that, I would (hand-wavingly) expect it can be made to work for a three-bit adder (with four bits of output)

There is valid use case for when designers simply need "fuzzy logic" or "fuzzy math" which doesn't need exact results, but which can tolerate inaccurate results. If using fuzzy hardware saves resources (time, energy, die space, etc.) then it might be a valid tradeoff to use inaccurate fuzzy hardware instead of exact hardware.

For instance instead of evaluating an exponential function in digital logic, it might be quicker and more energy-efficient to just evaluate it using a diode as an analog voltage, if the value is available as an analog voltage and if some noise is tolerable, as done with analog computers.

Reply on news.ycombinator.com