Optical Transformers

Nope, they're just going to make the model 100-8000x bigger then try to double then quadruple the width of the optical transformer.

Need to compare this with custom silicon like Apple will be shipping. They already have the Neural Engine chip which can run Stable Diffusion, but eventually you could imagine casting a specific model instance to an ASIC (say GPT-3.5 or -4, today).

If most devices are replaced within a year or two then you get a pretty good cadence for updating your Siri model (and even more incentive for users to upgrade hardware).

It will be also at least 100x times (physically) larger, because optical wavelength is ~1000nm vs 10nm of electronic gate size. So much for personal devices.

>I'm trying to imagine what a world would look like where generative text models are commodetised to the point where you can either generate text locally on your phone, or generate GBs of text in the cloud.

Dead internet theory for one. Scalable spear phishing and scams. Scalable automated offensive hacking. SEO far worse than anything possible today. Mass manipulation campaigns.

Social interaction would also be strange. Every messenger and dating app able to automatically reply and suggest sophisticated messages.

You still have to store and move the model bits. I/O is gonna be a problem

move to denser forms of communication i.e. math notation

>Furthermore, if we view transformers as Hopf algebras, one can bring convolutional models, diffusion models and transformers under a single umbrella.

I wish I was smart enough to know what a Hopf algebra was or how it worked, because this sounds awesome.

Wait, so your thesis would've led to no FPU, GPU, hw TCP, etc. We just slowly and linearly chug along with a classic 8086 beautiful ISA.

Well, the beautiful (not so beautiful these days) ISA is always there for you, while we grab a library to use a suitable specialized processor and run our shit 1000x faster for 1/100 the power.

I think the better way to look at it is that the norm is a very heterogenous architecture with all sorts of purpose built accelerators. However, while Moore’s law was in play the general compute ate all the accelerators (hence integrated fpus, simd, etc). Now that it’s dead, we’re going to go back and this is likely the new normal unless we somehow unlock a new era of exponential growth (eg optical compute). However, at this time that seems less likely so we’re likely to see more of these accelerators. The challenge of course is scale - specialized accelerators by definition have smaller market niches which in turn limits how cheap they can be and the creativity of what people think to do with it (imagine what computing would look like if only supercomputers had floating point units to support numerical simulations).

Even a single CPU operating like that is an illusion. Your ISA is controlling all sorts of stations within the CPU and getting good performance often means understanding hazards between the parallel processing occuring across different pipeline stages and stations.

General-purpose hardware can never be as efficient as special-purpose hardware, unfortunately. All computation has a structure to it that is defined by the way the data flows into the operations, which in turn suggests some optimal physical organization. So you first perform the computation on the general purpose hardware you already have, until you have to do it so often it becomes advantageous to design and build a more optimized structure for it.

If you're looking at deep learning, the end game is probably hardware a bit like the brain: a hundred billion of specialized, low-clocked coprocessors crammed in a small space with only local communication to their neighbours.

Isn’t the solution better “mechanical” and language integrations for “off-cpu compute”?

CUDA and subsequent libs arguably provided this and caused a pretty massive acceleration in the amount of GPU-utilising code that got written. Now, CUDA might still be pretty awful to write, but if general-propose-GPU API’s continue to evolve and refine, we might be able to get to a point where something like an LLVM compiler can auto generate efficient GPU code for matrix ops, much like how it can auto-vectorise certain patterns now.

"beautiful isa"? you've seen x86, right?!

I don’t understand this “stick to your guns(CPU)” mentality.

With Moore’s law dominating any ideas about software optimization, the mantra of “wait until the CPU is twice as fast in two years” has dominated the industry for long enough.

Not everything needs to fit in an idealistic CPU and we should allow for more complex computational machinery.

Let’s allow for new hardware, new programming paradigms, new ways of thinking.

I guess you dont like arms or legs then. We should all look like amorphous gelatinous blobs of undifferentiated flesh. Look, I have my own quirks - but you gotta squash those ones that make no sense.

I think more and more Channel I/O like objects is the future. Add on processors (service processors if you may) are part of that idea.

The key idea is you can use physical effects such as attenuating light to implement multiplication in a way that uses much less energy than the fairly complex arrangement of digital logic necessary to implement the same operation. This is roughly the method they use in the paper:

https://en.wikipedia.org/wiki/Spatial_light_modulator

I've seen a lab bench prototype of a different implementation, there are a lot of engineering problems to solve but as the paper points out the potential payoff is big.

Edit: The other key point is that one of the expensive components in transformers is effectively a giant matrix multiplication which implies many, many individual multiplications.

To run an AI model, you do not need a general-purpose computer, you don't need a powerful CPU capable of executing a wide variety of code. You mostly need massive amounts of matrix multiplication, and some simpler operations.

You also don't need very high precision: many models perform well even using 8-bit floats. This allows to ditch the whole digital approach and implement analog circuits, which, while sometimes less precise, are massively simpler and more energy-efficient.

So they built a mostly-analog device specialized for running ML models, and used optics instead of electronics, which allows to make certain things much faster, on top of the simplification.

There are known attempts to use electronic approaches, such as using flash-like structures that store charge as an analog value to store model weighs, and to do addition and multiplication right inside the cells.

There are lots of fun things you can do with optical circuits. One of the most fascinating to me was running multiple threads of execution through the same circuits at the same time at different wavelengths of light! I would speculate that there are similar sorts of things going on in optical transformers, but honestly trying to get deep enough into both the ML architecture AND the optical architectures implementing them is probably a month of spare time that I don't have at the moment :-).

I know much less about the large transformer models than I do about the optical hardware but the hardware ideas have been floating around since the 1980s. The development of lasers-on-a-chip made it all a lot more realizable. E.g.

"On-chip optical matrix-vector multiplier for parallel computation" (2013)

https://spie.org/news/4932-on-chip-optical-matrix-vector-mul...

Figure 1 has a nice visual of the vector-matrix computation in terms of the diode laser array, the multiplexer, the "microring modulator matrix" and the resulting output, the new (output) vector detection system.

> "We have designed and fabricated a prototype of a system capable of performing a multiplication of a M × N matrix A by a N × 1 vector B to give a M × 1 vector C. The mathematical procedure of MVM can be split into multiplications and additions, which is reflected in our design. Figure 1 shows a schematic of the architecture we propose. The elements of B are represented by the power of N modulated optical signals with N different wavelengths (λ1, λ2, …, λN), generated by N modulated laser diodes, either alone or together with N Mach-Zehnder modulators. These signals are multiplexed, passed through a common waveguide, and then projected onto M rows of the modulator matrix by a 1 × M optical splitter. Each element aij of matrix A is represented physically by the transmissivity of the microring modulator located in the ith row and the jth column of the modulator matrix. Each modulator in any one row only manipulates an optical signal with a specific wavelength."

That was 10 years ago, no idea what current state-of-the-art is.

I haven't read this paper yet but I'm familiar with the general work. The aspect that everyone ignores is that, yes linear transformations like matrix operations or fourier transforms are incredibly fast in optics, however the nonlinearity is the sticker. While optical propagation is nonlinear, you need very high intensities. The elephant in the room is that the linear operations rely on parallelism, i.e. they split the optical power up into multiple paths so each path has very low intensity, thus exhibits low nonlinearity. The solution that has been that everyone simply used optical to electrical conversion and did the nonlinearity digitally (or sometimes in analog electronics). That sort of works for one layer, but completely falls apart for multiple layers, it is neither cost not energy efficient to have hundreds or possibly thousands of a/d converters.

It's interesting because of the scaling law. No matter how much acceleration matrix multiplication gets on an electronic circuit, its energy usage is always going to scale as O(n^2.something). The implication here is that the energy usage by doing it optically is O(1). At least, that's how I read "We found that the optical energy per multiply-accumulate (MAC) scales as 1/d where d is the Transformer width". The best you can hope for is to stay on the right side of the constant factors (which, currently, the GPU world is).

Google audio search not only does speech to text, but can give you the song if you hum. I was with some friends who tried it outside in SF and it was a noisy environment on the sidewalk, and it was able to identify the song from their hums. Magic.

My understanding is that modern neural networks aren’t much like the brain, but machine learning researchers like the way it sounds to suggest that they are.

If the time between successive GPT releases halves after each successive release, we'll have to do what Bungie did with the third game in the Marathon trilogy and call it GPT-∞.

And then the post-singularity open source release, GPT-ℵ₁

GPT-5 level models could very well be distributed, training on millions of internet connected iphones. This is all pure speculation, we don't even know know what GPT-4 will look like.

And GPT-7 will be built with sticks and stones.

There are significant differences, because proof-of-work crypto mining is inherently a digital logic operation, so it's not amenable to photonic analog computing used for, e.g., approximate matrix multiplication and summation used in neural nets.

That said, photonic gates for digital circuits hold some promise for power efficiency and speed in their own way. For many of the parameters which apply to crypto mining, it's not so clear whether photonic digital circuits will significantly outperform the best near-future electronic transistors though.

(Source: I worked on a photonic crypto circuit design based on 4-wave (non-linear) mixing. Non-linearity is required for digital logic. Even though 4-wave mixing is energy efficient, the design was less efficient and performant than I'd initially hoped from the high spatial parallelism and basic THz figures. The principles still hold a lot of promise.)

Seems unlikely. The paper is about performing certain computations in the analog domain, trading precision for power efficiency. Hashing requires lossless computation.

PoWx.org or something else?

Sounds interesting, can you say who it is?

It doesn't train, weights are pre-programmed and it executes.