Why not faster computation via evolution and diffracted light? | Hacker News Reader

Why not faster computation via evolution and diffracted light?

interconnected.org

67 pointstobr32 comments

I think this is inspirational but it seems like puffy popular science ("pseudo-science" may or may not be too strong?). I'll quote what I wrote to a friend about this article:

> It was intriguing and piqued my interest. However, it smells a lot like hyped-up crackpot science to me ... if it seems too good to be true, it probably is. There's a lot of vagueness here, "what ifs", etc. Like "What if we could evolve hardware to make use of hidden physics?" Yes, well, what if? What is "hidden physics"? And why doesn't the author try it? :-)

> I looked briefly at his bio, and his background is "new media" and "addressing abstract social and technological ideas to mixed audiences" ... (computer) science, not so much.

> That said, I do like the idea of reducing abstraction layers, getting closer to the hardware, and using seemingly-strange physics for what it's worth. I'm just very skeptical of his framing of it as something which could make everything 1000x faster overnight.

> I remember reading a book years ago by "futurist" Michio Kaku called Visions. At the time I was really inspired by his "visions", which sounded very scientific. However, I think they were probably just vague, well, visions. This writing strikes me as similar.

> What is "hidden physics"?

Physics we don't know about. Our cells are doing that as we speak – though we'd probably call most of it “hidden biochemistry”. I think it unlikely that we'd stumble upon such “hidden physics” unless we were evolving objects at close to the atomic scale, but it's possible; perhaps there's some way of getting phonons to interact with each other that behaves like Brownian motion, and then you can get metaphonons?

The author doesn't try it probably because the author doesn't know how. I don't try it because I suspect it's not possible with the technology I have access to.

In the realm of ideas about possibilities, isn’t his bio is somewhat irrelevant unless his background is in science fiction? I wouldn’t underestimate the power of good science fiction in inspiring science itself, or in the case of this essay/thought-piece, of inspiring existing and future research into alternative computing platforms and/or new paradigms of software execution on existing platforms. On my part, it reminded me to look back into FPGAs and weight-agnostic neural networks.

> That said, I do like the idea of reducing abstraction layers, getting closer to the hardware,

Abstraction is THE tool to curb complexity. It's a feature that allows us to design, reason about, build, debug, maintain and extend systems with the complexity of todays computers. Reducing abstraction is in itself not a desirable goal.

What could be useful is to reduce the overhead introduced by abstraction. Doing that by removing the abstraction itself is only one such way and a quite costly alternative.

I have no idea what the author is trying to say.

Given that they "just recently" learned about microcode, I'd not hold my breath expecting a profound insight about computer architecture or anything computer science though.

Seems to be asking for some sort of optical FPGA thing with the hope that light will be immensely faster than electricity. Maybe not knowing that it's not?

Also some hints at wanting this "optical FPGA" thing to be analog, rather than digital. And some notion that the various layers of abstraction (silicon->microcode->asm->code) must be wasting lots of cycles.

I'm not really clear on where he thinks 40 years of instant fast-forward might come from.

If you look at some of the earliest Synthetic Aperture Radar (SAR) systems, you'll find that they used exposed film (i.e. light) to perform some serious fourier-style computations using diffractive optics. I.e. you can use physical processes to perform useful computations. It's not entirely out of the realm of possibility... it's just that silicon & Moore's law generated a lot of deterministic, general-purpose compute capability -- whether CPU or GPU.

it's kind of akin to certain compute jobs using quantum effects: There are a class of algorithms that can benefit from that kind of co-processor.

Off the cuff, I don't know of any compute jobs where optical "computers" give you major speedups compared to standard digital processes. But it's not altogether crazy.

> Seems to be asking for some sort of optical FPGA thing with the hope that light will be immensely faster than electricity. Maybe not knowing that it's not?

But really it is. Except for interconnects or analog RF stages, ICs aren’t built as if they were solid states waveguides. Indeed speed is determined by how powerful the charge pumps are in moving electrons, not quite the speed of EM propagation

He's hoping for "a million times faster", and "that it might be possible, with today’s technology".

You can certainly go back to analog for calculating ML. In a way, that’s what Rain Neuromorphics is doing.

On the one hand:

> Could that task be performed by simply the right set of transistors, at the hardware level, no matter how insanely arranged?

> What shortcuts could be taken?

Seems to be suggesting ASICs as is done with Bitcoin mining. On the other about optical rather than electrical which could be more efficient, i.e. produce less heat.

I was expecting a completely different post about evolution of brains that operate with photons, and another jump to entangled ones.

but that's basically already solved with FPGA, right? You can create any circuit on the fly with code.

They propose to implement an algorithm such as a trained neural network by an single-purpose analog computer (whether electronic or optical).

This is sound idea and subject to research but he himself lists some disadvantages and I can imagine there are many more. Generally, nonlinear analog systems are VERY fickle.

> G enerally, nonlinear analog systems are VERY fickle.

You have used a whole bunch of non linear analog systems to write this text..

Of course all existing systems are analog, I know everything digital is just a pretension. But that does not help us in this discussion.

So...I am sure you know what I mean, can you please help me to express it better?

> Let’s say you just wanted to perform just one task. Say, recognise a face. Or know whether a number is prime or not. And you didn’t care about flexibility at all.

> Could that task be performed by simply the right set of transistors, at the hardware level, no matter how insanely arranged?

For quite a lot of things .. sort of yes. You can do functional reduction. Then you get a chip that does exactly one thing. Which works great until you want to change it. Since change flexibility is incredibly important, that's why we've gone the opposite direction and use microcontrollers for things that could be done with ASICs.

He misinterprets Thompson though; the strange physical effects of the evolved circuits aren't "hidden", they're just outside the model. We rely on simplified models to make behavior predictable, and we design circuits to be as modellable as possible and not use the properties that are hard to model.

> the strange physical effects of the evolved circuits aren't "hidden", they're just outside the model

But isn't that all the same -- feels small difference to say it's not "hidden", it's just "outside the model". Go far enough outside the model, and it's as good as hidden. (as opposed to simply going a little outside the model, which is avoided as a mild preference)

For example, we don't understand many aspects of the biokinetics of neurotransmitters binding to the active sites of receptors, but the assumption is that in aggregate, things like this are important to our neurochemistry. And this, a million times over for all protein interactions. Things like this are literally still not modellable in a way that's useful to predicting emergent behavior, but despite that, they play a part in how we experience stimuli. For all intents and purposes, this is still "hidden" to us in its complexity. But the environment has nonetheless "selected" (in the evolutionary sense) on the cumulative results for a million years.

Sorry, studied biochemistry, so apologies if this seems non-sequitur. It feels quite related to me :)

I don't get the negativity. This is indeed an emerging field, so just going to throw in a few examples:

- Google using RL for chip layout optimization [1]

- Stanford profs optimizing lenses end to end with image processing [2]

As for the "diffractive deep neural network" paper it is not really a deep network as they don't implement the non-linearities in the optical domain. That still is the hard bit, so maybe the monocle is not realistic, but fortunately there are ways it can still speed up computation.

There is a general consensus that GPU-s enabled the current deep learning revolution and perhaps as argued by the Hardware Lottery paper [3] backprop is really just the current lucky ticket. So why not try to evolve hardware with ML for other algorithms too?

As for the questions raised, I do want to know your answer to this:

> a question for computer scientists, what single question would you ask if you have a dedicated computer that was [that multiplier] faster?

[1] https://ai.googleblog.com/2020/04/chip-design-with-deep-rein...

[2] https://www.youtube.com/watch?v=iJdsxXOfqvw

[3] https://hardwarelottery.github.io

>I don't get the negativity.

I suppose my response looks negative. What drove that was that the writeup seemed to be "high confidence in something very unusual[1]" coupled with "very little detail on what he thinks would accomplish that[2]".

[1] "Million times faster" "with today’s technology" "40 years of performance gain"

[2] I gathered only analog vs digital, light instead of electricity, and some sort of analog/optical FPGA.

I felt negative only in the sense that the confidence seemed very high, but I couldn't see any detail that seemed to support it, or enough detail to search elsewhere. Especially with all the references that we could do it now.

> a question for computer scientists, what single question would you ask if you have a dedicated computer that was [that multiplier] faster?

Assuming climate change is the largest threat looming in the near future, I suppose better modeling and prediction on what to do, and when.

Speed and efficiency are often lost to gain flexibility, and this tradeoff is often ignored when you see articles extolling non-human intelligence of birds, insects, etc. Insects, through evolution have created highly efficient and FAST neural circuits that lead to fit behaviors, but the fitness of these circuits and behaviors often mask their inflexibility to a change of context. Change the context and the lightning fast reflexes of a fly can be made to make it unfit.

This is a really great point. The difference is between instinct and intelligence. But is the difference merely due to a greater amount of neurons between a spider and a chimp, or is there a fundamentally different architecture of those neurons that enables intelligence over instinct? Not coincidentally, the creatures we most often associate with intelligence happen to be mammals, who alone have a neo-cortex.

Regarding lack of mental flexibility - it’s like when you move a turtle to a new patch of grass. Sometimes they get confused by the teleportation and will stop eating until they’ve made their way to their home range, which is sometimes never.

The thing about circuits is they are all an approximation of Maxwell's equations. Maxwell's equations like all wave equations have analytic solutions (in the mathematical sense of the world 'analytic'), which extend to infinity in terms of their interaction and sensitivity in an absolute sense.

Digital circuits are an approximation to this approximation. But the underlying reality of Maxwell's is still there. There is NOT black magic; only lack of understanding.

When you do "analog circuit analysis", you MUST solve the entire circuit holistically as a system of equations which cover every physical corner. Because of this analytic solution reality to wave equations. (And yes, Schrödinger's is also a wave equation and this is the reality of origin of "interaction at a distance").

So strictly every component matters but it's to some degree of sensitivity and accuracy as specification. Taking any component out, if accuracy matters to all digits past the decimal point, will ALWAYS have an effect on the operation and performance. But engineering is about knowing how many digits past matter and how many do NOT. And there's no strict definition to go by - "it depends" mostly on the situation and requirements.

I guess the author of the article isn't familiar with this underlying reality of circuits and the physics behind them. Or the author is overselling what is trivial for signaling something other than truth or the facts.

I think the part they are missing is that a general purpose computer is required to train and simulate the physical model. The speed at which a trained model executes isn’t really a problem in most domains as far as I’m aware. They can evaluate in real time already on standard hardware. It’s also more efficient to push software updates when your model improves rather than retooling you manufacturing process and deprecating valuable resources into a landfill.

Huygens Optics on Youtube [0] has some interesting research into optical gates using interference.

[0] https://www.youtube.com/watch?v=pS1zAAD1nXI

The negativity here reminds me that there is a spectrum among engineers: between high-level and low-level thinking, between goals and means, between abstractions and operations, between simplification and optimization. Most sit somewhere in the middle, but if you put two people on opposite sides into a discussion, you’re going to see fireworks. “My implementation runs 10x as fast” verses “My implementation is maintainable.” Both types of thinking are necessary, but you need to know where the other person is coming from.

This essay is a thought-piece, exploring possibilities. It isn’t a specification for a new form of computation. It’s an imagination of new possibilities.

And as such, it has a lot of merit.

Take a look at “Weight Agnostic Neural Networks” by Adam Gaier and David Ha at Google Brain: https://weightagnostic.github.io. Their research shows that very small RL neural networks can be evolved with shared weights to solve control tasks. My immediate thought after reading the paper was that an FPGA would be well suited for this kind of model — certainly at inference, but possibly at training time considering that the model is trained without backprop. The limit on training becomes the latency in updating the FPGA during the training loop. Moreover, you should be able to run many versions of a model on a single FPGA by dividing it up into batches.

If there’s one thing we know about DL, it is that computational power always wins (The Bitter Lesson - http://www.incompleteideas.net/IncIdeas/BitterLesson.html). You can tackle that in two ways: increasing compute or increasing compute efficiency. The Transformer started out as the latter: replacing sequential RNNs with parallel Attention. It seems likely that the next big breakthrough will also find a way to get 10x computation out of current hardware.

Maybe the idea of non-differentiable highly-customized networks will be that breakthrough. Maybe the essay is right that we’ll find the breakthrough with a new paradigm in hardware itself, Maybe not. It’s worth thinking about, if only because it is so fascinating.

The whole space of analogue computing (circuits) did get a big boost in recent years (people had been doing it for years, but it never amounted to anything), largely because of the requirements of deep learning, which is essentially large matrix multiplications. I guess many of the recent ML accelerators fall under this category in some way.

There's also been quite a bit of work on using optics (mainly integrated optics) for these tasks, but it's still very open if that will amount to anything.

All that said, I really would be hesitant to call this computing in the traditional sense, it's more like an accelerator card if anything.

I thought immediately of LightOns work: https://lighton.ai/photonic-computing-for-ai/

And this is very much already a vetted approach. They use photonics circuits to implement random projections, which can be used to speed up t linear algebra in modern ML workflows. It’s no magic monocle, rather a pretty interesting co-processor racked away in some data center.

One day someone dreamed that he could fly like birds. Today we fly much higher and faster than them. Scientific achievement always begins with the imagination.

We don't fly like birds. Evolved designs find local maximums and stop.

To your point, Jules Verne imagined shooting men to the moon with a big gun. But to OP’s point, the vision inspired enough people that we eventually built rockets. There’s no dichotomy here. But there is a dependency: means for nowhere without ends, and the ends often generate the means (so much so that we speak of ends not justifying means).

I think they're on to something. Our constraint on abstraction layers that they must be legible and grokkable by human minds, that's just something we impose because we feel that we should understand. But that constraint is overburden. I feel that neural nets are starting to prod at this. Our biology works and does incredibly complex computation (including social computation), and it's creation process didn't give a fuck whether a human could understand it. We're still playing catch-up with that sort of low-power computation that biology does. A sandwich powers a wildly powerful neural network

This also reminds me of how alien and organic ML-assisted design is -- when a design task is constrained only by its environmental constraints, and human aesthetic and comforts are discarded: http://jcsitnet.com/journals/jcsit/Vol_7_No_2_December_2019/... https://medium.com/intuitionmachine/the-alien-look-of-deep-l...

Straight lines and clean edges and nice angles are just abstractions that we humans happen to have preference for due to our very particular evolutionary history. As in, we happen to like clean straight lines and certain logics in physical and visual domain, but have no such constraints on taste and smell preferences, and in fact love complexity in those sensory domains, because that served us in our past ecological niches. But discarding these visual/logical sensibilities and allowing abstraction interfaces to float free of our preferences, that could open up a ton of possibilities for improvement. Of course, it would become as magical and messy as biology, and perhaps quite challenging to our reasoning.

There's good sci-fi about this. About our machines having more comfort with these complexities, and moving into realms of work where we just can't keep up. And we either trust-fall into being ok with what they're building, or we cripple their abilities to match ours. And maybe the society that cripples its AI less, ends up running circles around the ones who do...?

EDIT: to be clear, I believe our minds are already doing this: our brain is using "hidden physics" of matter to bring about our conscious experience. We keep finding that ever more niche effects are potential important to the hardware that runs our minds. Because "matter" didn't give a fuck about being legible to its future inhabitant :) We're still teasing apart the interactions, and likely will succeed in regrowing consciousness (with additional legibility) before we actually fully understand how our own is working.

Discloser: Am trained as biochemist, so might have a bent on this that doesn't resonate with more hard CS folks.

Reply on news.ycombinator.com