Hacker News Reader

Comment by mikewarot | Hacker News Reader

TinyGrad is GeoHot's system/compiler to map neural networks onto hardware. He consistently points out this one point: Because the exact number of cycles is know in advance, it can be scheduled, there's no need for branch prediction, or that type of thing in a CPU.

Essentially, he wants to be able to express programs, and even an operating system, as a directed acyclic graph of logical binary operations, so that you can have consistent and deterministic runtime behavior.

The bit about LLMs is a distraction, in my opinion.

View on HN

> he wants to be able to express programs, and even an operating system, as a directed acyclic graph of logical binary operations, so that you can have consistent and deterministic runtime behavior.

So how is this different from digital logic synthesis for CPLD/FPGA or chip design we have been doing over the last decades?

FPGAs are (prematurely) optimized for the wrong things, latency and utilization. The hardware is heterogeneous, and there isn't one standard chip. Plus they tend to be expensive.

The idea is to be able to compile/run like you can now with your Von Neuman machine.

FPGA compile runs can sometimes take days! And of course, chips take months and quite a bit of money for each try through the loop.

With FPGAs I can sample a hundred high precision ADCs in parallel and feed them through DSP, process 10Gb ethernet at line rate, etc with deterministic outcomes (necessary given safety and regulatory considerations). They integrate well with CPUs and other coprocessors - heterogeny isn't wrong. Plus training a NN model also takes days! To be fair not always, but for the above applications my build time was hours to many-hours anyway.

I grant the hardware is absurdly expensive at the high end, but I really don't think application wise the comparison is apples to apples.

Hotzs saying literally everything with an io pin or actuator will be driven solely by NN (driven by tinygrad) seems to me maybe 1/3 self promotion, 1/3 mania, some much smaller amount incisive at best.

Reply on news.ycombinator.com