Hacker News Reader

Comment by Roark66 | Hacker News Reader

Personally I think we're doing AI wrong. Every time we simplify the architecture we get huge improvement in speed. CPUs have to everything (load/store, branching, execution), GPUs do a lot less branching and aren't optimised for it. This is why they can be made massively parallel. What if we cut out arbitrary load/store too? This would look like A DSP. All memory would be prepared in advance and the massively parallel accelerator would "stream walk" through It without having to load store arbitrary memory locations. The speed improvement could be on par with CPU->Gpu. Of course this would be for inference mostly. There is a startup that is trying to do just that called tinycorp. I'm certainly watching it with interest

View on HN

> All memory would be prepared in advance and the massively parallel accelerator would "stream walk" through It without having to load store arbitrary memory locations.

Dedicated accelerators do that already, e.g. google's TPUs, tesla's D1 or apple's neural engine. You must load the data into compute-unit local memory first before executing matmuls. Keeping the weights there and only piping the dynamic data through it saves memory bandwidth.

This sort of architecture only really works when the network is small enough, which has been a perennial problem for neural network accelerators as networks grow but accelerators don't. LLMs and the like will often prefer the opposite form of "stream walking" (streaming the weights through the data) or a hybrid.

Streams weights instead of data sounds really interesting - I had never considered it.

Something else that might be theoretically possible is -

Large array of FPGAs are apparently used to simulate and verify chips [1], can the same be done to run LLMs? Can we have 0.25 to 1 token per cycle, would the engineering effort be worth it, and would it be financially feasible from a TCO standpoint?

[1] https://www.servethehome.com/amd-vp1902-is-leviathan-fpga-do...

The only problem is that an array of FPGAs would make an Nvidia GPU look cheap. And easy to get.

People have used FPGAs for DNNs before

https://www.microsoft.com/en-us/research/project/project-bra...

FGPA’s have been tried and dismissed. GPU’s are simply better with speed and efficiency.

This sounds a lot like the thesis behind George Hotz’s tinygrad, and the accelerators he eventually hopes to design for it.

edit: typo

There have been some research projects in this direction (computing fabrics, in-memory computation) but nothing that I'm aware of that made it to larger scale manufacturing.

Itanium tried removing a bunch of the out-of-order hardware and hoped that compilers could schedule everything in advance. Generally, that did not work very well.

I don’t get the sense that this is what the parent comment is talking about at all.

Not to mention that GPUs already execute in-order (at least any that I’m familiar with). They do have multiple execution pipelines, but instruction fetch/decode is in-order unlike something like a typical modern high performance CPU.

As far as I know you can preload the data in the GPU before processing it. What is the difference/advantage of what you are proposing?

You wouldn't need the hardware to support arbitrary load/stores. In particular, you could get rid of (some of) the address lines...

I'm unsure if this would be much of a win.

well, the data is way smaller than the model (at least the current trend), and you probably still need random access for the weights of the model. I am not sure if it a gain worth to pursue.

Reply on news.ycombinator.com