The physical process that powers a new type of generative AI

Although arguably this could be why we only use a tiny fraction of our brains, with most of our memories staying in long-term storage, but I digress.

I first encountered a similar situation back in the early 1990s when I was trying to get my Mac LC II to blit a whole 640x480 screen at 60 fps in 8/16/32 bit color:

https://everymac.com/systems/apple/mac_lc/specs/mac_lc_ii.ht...

It had IMHO an artificially slow memory bus that was only 16 MHz and 16 bits, which is just over 100 fps in 8 bit color: (16e6) * (16/8)/(640 * 480) = 104.167. 16 bit color or higher at that speed was apparently unachievable. Also I didn't know how pipelines worked, so there might have been a way to interleave instructions to do a direct copy from RAM to VRAM or do it through DMA or set certain low memory bytes to access page flipping or do palette animation, etc. Apple hid all of that stuff (intentionally?) to prevent the Mac from becoming a gaming platform because Steve Jobs never loved that direction.

So anyway, I learned early on that all that matters is the memory bus, not the FLOPs. That got cemented when I was taking ECE classes in college and learned how von Neumann architecture tends to reinforce the idea of having a central bus. I rejected that, and began to dream of a parallel architecture (probably MIPS) where every CPU is part of a mesh with at least a 4 way connection to its neighbors, although today I might do CPUs in layers with vias to provide 6 way. Or abandon that approach and use a 2D grid of busses where each CPU shares bandwidth and its memory controller makes hashes for a content-addressable memory, starting with something simple like CRC32 and falling back to a more collision-resistant hash. So that the machine could be programmed the same was as the web rather than relying on elaborate caching. But the OS would present a desktop programming style single unified address space to the kernel, protected for each process, with say 256 or 1024 cores to start, but easily scaling to 1 million cores or unlimited if the machines are connected with gigabit ethernet or similar. Users could run something like Docker to mimic environments they're used to like AWS EC2. Then run something like MATLAB or Julia directly to have the power of GPU shaders but in a MIMD fashion rather than SIMD. It's all pretty obvious and I've been ranting about it online for a decade or two, so at this point I've resigned myself to it never happening. But RISC-V gives one hope.

Anyway, I dreamed about a world where kids would have access to such hardware to get real work done. Rather than having to learn the intricacies of GPU programming through middleware like OpenGL and its successors. If we had a billion of those machines around the world in 2023, I truly believe that we wouldn't have to rent AI learning services in the cloud.

But that never came to be, and now my heartfelt platitudes just feel like projection. Like I'm casting shade on Nvidia and others and I don't mean to do that.

So to rephrase my parent comment: I'm more concerned with training than I am with the performance of trained models. So I wanted the tech industry to improve the hardware to the point where these algorithms are trivial, rather than forcing users to manually consider the nuances of the algorithms in a bare hands fashion at great expense. My approach would have been to have genetic algorithms train and run their own genetic algorithms to evolve neurons and the topology of the neural net collaboratively on the web over many generations. More like artificial life than machine learning. IMHO this simply can't be done with SIMD GPUs, so I never would have started down the road we're on today. But people are making fortunes on the status quo, so it's too late for me and I just sound like a misanthrope, and that's ok!