Hacker News Reader

Comment by hesdeadjim | Hacker News Reader

I find it so cool that technology created to make games like Quake look pretty has ended up becoming a core foundation of high performance computing and AI.

View on HN

I think it's even cooler how matrix multiplication dominates both the universe at large, and the systems that understand it (neural networks).

Well a large portion of that is desiring the data to be in that form. BLAS operations are ruthlessly efficient and use the system hardware so well.

Relevant article:

https://www.technologyreview.com/s/602344/the-extraordinary-...

Max Tegmark keeps saying interesting things lately. Controversial for sure, but interesting. His book 'Our Mathematical Universe' (related to the article you've linked) is thought-provoking, and I would label it a must read if you're interested in what's going on at the outer edges of fundamental science. The chapters are clearly separated into: fact, hypothesis, and far-out speculation, so there's no need to criticize the whole thing indiscriminately.

There was a series of attempts by Lee Smolin and others to come up with a theory of quantum gravity by assuming that the universe, at the bottom, is essentially simple and discrete (not in the fixed-grid sense, but in the sense of a discrete web of relations). That model also exhibits a remarkable similarity between the structure of the universe, and the structure of the neural networks that understand it.

The future of fundamental science is sure to be fascinating.

I think that's a dreadful article. We do know how neural networks work; they're a bunch of hierarchical probabilistic calculations that are pipelined. I don't really see how that couldn't work well; it's just hard to find the right probabilities. The difficulty is far more in the training than the working, and that's where the deep learning advances come in - inferring more parameters in a deeper hierarchy.

There's no relationship between a hierarchy of probabilistic estimations and a hierarchical decomposition of the cosmos. The cosmos forms an apparent hierarchy because of the rules that govern matter and the initial expansion of the universe. That a small number of parameters might be listed in describing both is neither here nor there. A small number of parameters describe the vectors in a font file. It doesn't follow that a typeface then has any relationship with my brain or the universe.

The article reads, to me, like this: neural networks are this cool hierarchy thing, the cosmos is this cool hierarchy thing, and both of these things have low Kolmogorov complexity, isn't it amazing that our brains are like this and can understand the universe, wow.

> a bunch of hierarchical probabilistic calculations that are pipelined

That's one way of describing quantum theory; generally "contextual" or "non-commuting" are used instead of "hierarchical".

If the universality of such a common framework doesn't seem profound to you, at least realise it isn't something generally appreciated and barely even hinted at just a few decades ago.

Linear algebra is the ultimate common variable in technical computing and applied mathematics at large.

I find it incredible that even with all these cool applications of matrix multiplication it gets taught so horribly in schools.

And yet taught so well by Gilbert Strang and Peter Sheridan Dodds or Math The Beautiful:

https://www.youtube.com/playlist?list=PLE7DDD91010BC51F8

https://www.youtube.com/playlist?list=PLX5cCY3jYXJSDzX_lszck...

https://www.youtube.com/channel/UCr22xikWUK2yUW4YxOKXclQ/pla...

It backwards: first common application of the technology gave it the name.

Like in one of the Stanisław Lem's stories about Ijon Tichy people call intelligent anthropomorphic robots washing machines.

You'll be delighted to hear that traffic signals are called "robots" in South Africa.

That itself reminded me, that the word "robot" comes from "robota, rabota" which means "to work, worker" in a lot of slavic languages (My native language is bulgarian).

From the dictionary:

robot, origin: from Czech, from robota ‘forced labor.’ The term was coined in K. Čapek's play R.U.R. ‘Rossum's Universal Robots’ (1920).

https://www.google.com/search?q=robot+definition

Rossum is a riff on "rozum" which means reason (as in to reason about).

https://en.wikipedia.org/wiki/R.U.R.#Origin_of_the_word

"Thinker's Universal Workers"?

Yet another step in the progression in which mass market GPU silicon kills traditional vector and memory bandwidth rich4 HPC/supercomputing hardware. Cray-on-a-chip.

Edit: traditional vector machines like the nec sx still hold the programmability crown because you get a usable single system image, right?

Matrix multiplication is important for graphics and important for finding the weights of a neural network

Yep, hard to imagine though that the original creators of the Nvidia TNT or Voodoo had any idea that GPUs would become fully programmable computing hardware used for non-graphical applications.

I don't think they'd have been super surprised. Just pleasantly happy.

AI Accelerators have been a thing for decades - DSPs were used as neural network accelerators in the early 90s - and Cell processors were a thing by 2001.

GPUs just became vastly more accessible to general purpose program in the last decade. People were doing it back in the 90s but it was seriously hard.

We finally hit a tipping point where it's just kinda hard.

There were also the various custom "systolic array" processor designs in the 1980s (the ALVINN vehicle, and earlier projects which led to it, used these for early neural-network based self-driving experiments).

I remember back in 2004 when I heard a fellow grad student was working on using GPUs as a co-processor for scientific computing, I though "Wow, that's esoteric and niche."

This reminds me of a comment i read here ages ago about a scientist using the "processors" of the univercity's postscript printers because they did the work much faster than their scientific workstations.

Reminds me of some Commodore 64 programs running code on the 1541 disk drive to offload computation from the main CPU (both the C64 and the 1541 had 6502s (well, the C64 had a 6510 which had an I/O port) running @ 1Mhz). The original Apple Laserwriter had a 68k running at 12Mhz, while the Mac Plus, which came out almost a year later, had its 68k running at 8Mhz.

Creators of Voodoo (3dfx = Gary Tarolli, Scott Sellers) came from the world of fully programmable GPUs. Silicon Graphics workstations had full T&L since ~1988 (http://www.sgistuff.net/hardware/systems/iris3000.html).

The whole point of Voodoo 1 was making it as simple and cheap as possible by removing all the advanced features and calculating geometry/lighting on the CPU.

https://www.youtube.com/watch?v=3MghYhf-GhU

Iris Graphics Geometry Engines weren't programmable in the modern sense. There was a fixed pipeline of Matrix units, clippers and so on that fed the fixed function Raster Engines. You could change various state that went into the various stages, but the pipeline's operations were fixed.

Later SGI Geometry Engines used custom, very specialized DSP-like processors, but the microcode for those were written by SGI, and not end-user programmable.

There were probably research systems before it, but AFAIK the Geforce 3 was the first (highly limited) programmable geometry processor that was generally commercially available.

Uhm, weren't their later graphics systems heavily based on i860 processors?

Yes, later REs were i860s.

https://www.youtube.com/watch?v=ooLO2xeyJZA

Reply on news.ycombinator.com