Hacker News Reader

Comment by j1vms | Hacker News Reader

I wouldn't be surprised if Google is looking to build (or done so already) a highly dense and parallel analog computer with limited precision ADC/DACs. I mean that's simplifying things quite a bit, but it would probably map pretty well to the Tensorflow application.

View on HN

Computing with opamp primitives has somewhat fallen out of favor since you've been away..

The primary cost factor in anything computing is just power. You can always buy more of the things, but power is the ongoing cost and every watt in computing costs you extra in cooling power.

What are the advantages of analog computing for this application?

If you have an application that can tolerate error (like classification), then analog computing can give enormous gains in terms of speed _and_ power efficiency. Essentially, the savings come from using physics to perform the math (see Kirchhoff's current law) vs. using discrete time steps vs. fully-unrolling the logic. Google may not be using analog processing for this version, but I read an analog neural network researcher's page who said he moved to Google last year. (Sorry, I can't find the page again, but I think he was from the UK.)

What do you think about http://optalysys.com/ or http://lighton.io?

I think Optalysys looks interesting!

For the curious, Optalysys has built a general purpose optics-based correlation/pattern matching machine. From some of their predecessor-company marketing material: The correlator performs pattern matching on large data sets such as high-resolution images, providing a measure of similarity and relative position between objects within the input scene. This allows large images [and general data converted to images] to be analysed far faster than electronic equivalents.

Going back to the topic of NN-based computing, I found this talk to be intriguing: https://www.youtube.com/watch?v=dkIuIIp6bl0. The main argument is that because Moore's law may no longer be in effect, it will become increasingly important to explore alternate computing solutions. (Google's TPU could be supporting evidence for this argument.) The speaker also co-authored a paper which I liked "General-Purpose Code Acceleration with Limited-Precision Analog Computation".

Thank was a fantastic talk, thanks for the link.

This was a fantastic talk, thanks for the link.

This video was very cool. Are there any IC's that can perform analog computing for neural networks on the market now? I'm picturing something like an FPGA but with a bunch of op amps that you can connect into summers or amplifiers.

If not, how would one practically implement an analog computer for neural network programming (without several tables full of op-amps?)

Glad you liked the video!

You can implement an analog neural network yourself using a Field Programmable Analog Array. (I've never done it, but you'll see academics online writing papers about it.)

Another thing that is sort of related is Lyric Semiconductor; they built these cool application-specific probabilistic processors; they were purchased by Analog Devices a while back.

True. ML applications are well-suited to analog computation not only because they can tolerate errors, they also have an ability to adapt to errors, provided training algorithms are ran in hardware.

Analog circuits run at full-speed (no clock), use little power, and take up little space. An analog computer directly implements the mathematical function it represents as circuits instead of emulating it on a von Neumann-like machine. Long story short, they have issues that made people go digital. Yet, if you can use analog, you can get significant advantages. Example for math acceleration:

http://www.cisl.columbia.edu/grads/gcowan/vlsianalog.pdf

Brain is a bunch of components that are spread out 3D that operate like a mathematical function at slow speed. Mostly sounds analog. Results in us. So, a huge spread of analog components could get some results directly simulating something like that. Here's one of my favorites which is a wafer-scale, analog computer for neural networks.

www.kip.uni-heidelberg.de/Veroeffentlichungen/download.cgi/4713/ps/1856.pdf

Reply on news.ycombinator.com