Tinygrad 0.9.0 | Hacker News Reader

Tinygrad 0.9.0

github.com

241 pointswozeparrot48 comments

I see they have experimental AMD backends that don’t use ROCm. Is ROCm that bad that they wrote their own, or was there some other justification for this?

The author of the library has done numerous YouTube rants on how bad he thinks the AMD compute drivers are.

What does he get wrong?

He's not wrong.

Has he? The ones I'm aware of he was complaining about the low quality of the generic kernel drivers. AMD software had a tendency to crash when doing anything outside of standard video games (which was my experience too, but I've caved and bought Nvidia since then; average driver quality of Nvidia on linux seems to be much lower but the kernel doesn't go down which is nice. Got a lot of OOM errors where on ROCm the kernel froze requiring a full system restart).

But this is interesting and probably strong evidence that the CUDA API isn't the moat people thought it was. CUDA multiplies matricies and that is close to a commodity operation. The moat actually seems to be Nvidia's higher generic software engineering standards, the difficulty in writing job scheduling/memory management infrastructure and possibly the fact that closed firmware is the norm.

Arguably the nvidia AI moat is PyTorch and the heavily optimized libraries behind it. The CUDA language and toolchain helped get that effort off the ground, no doubt, but PyTorch is written and optimized for CUDA first. All other backends work best with similar semantics to CUDA and have to match Cuda semantics to keep their users happy.

Yes he has. I have seen multiple episodes on his YouTube[1] where he absolutely grills the whole company. He also gave them a deadline to opensource the drivers or he would stop trying to make AMD stuff work.

Sorry for no direct link, but he has so many and very long videos that it is hard to find the exact spot.

https://www.youtube.com/@geohotarchive

Seems to be an issue on their side. E.g., for a step of GPT2 training on a 7900 XTX [1]: tinygrad is ~440ms, PyTorch 2.4.0.dev20240513 is ~97ms, Karpathy's llm.c with ROCm is ~79ms, and llm.c with custom kernels is ~58ms

[1] https://github.com/anthonix/llm.c [2] https://github.com/tinygrad/tinygrad/issues/4301

That issue seems a month old, while the 58ms number looks 1 day old.

I have seen last month getting a lot of work done in improving performance (it's in the release announcement as well), but of course I still don't think it can compete with that number...still, a new comparision would be cool.

Ran tinygrad again about a week ago, no change.

And still no comment on the issue, will re-run if there is any comment.

Thanks for the answer

Tinygrad targets consumer hardware (to be precise, only Radeon 7900XTX and nothing else[1]), while ROCm does not actually provide good support for such hardware. For example, last release of hipBLASLt-6.1.1 library has deep integration with PyTorch[1], while working only on AMD Instinct hardware. And even for the professional hardware out there, the support period is ridiculous: AMD Instinct MI100 (2020) is not supported. Only 4 years and tens of thousands of dollars worth of hardware is going to the trash, yay!

And to be more precise, they still use some core libraries from ROCm stack[3], they just don't use all these fancy multi-gigabyte[4] hardware-limited rocBLAS/hipBLASlt/rocWMMA/rocRAND/etc. libraries.

[1] https://tinygrad.org/#tinybox

[2] https://github.com/pytorch/pytorch/issues/119081

[3] https://github.com/tinygrad/tinygrad/blob/v0.9.0/tinygrad/ru...

[4] https://repo.radeon.com/rocm/yum/6.1.1/main/

Even without hipBLASlt, PyTorch is still ~4x faster than tinygrad on a 7900 XTX for GPT2, and works fine. Any idea why?

Maybe due to 2x slower matmul (well known bottleneck), as one of multiple factors? https://github.com/tinygrad/tinygrad/blob/v0.9.0/extra/gemm/...

There are multiple bounties just for it in https://docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic...

I think the matmul issue is symptomatic of a much deeper issue.

It would be nice to see less whining and blaming AMD (PyTorch and llm.c actually work on 7900 XTX, and blow tiny grad out of the water in terms of perf!), and more just getting stuff to work.

The idea with the experimental backends is that they will talk directly with the kernel drivers, [3] is for the HSA backend but is not needed for the AMD backend.

You might consider posting not the release page, but rather the repo page:

https://github.com/tinygrad/tinygrad

Another good alternative would be:

https://tinygrad.org/#tinygrad

That's been posed many times*. What's new here is version 0.9.0

* https://hn.algolia.com/?q=https%3A%2F%2Fgithub.com%2Ftinygra...

And yet, I never encountered any of those other posts. I bet that's true for most HN readers.

But, suit yourself. I just think you'll get higher engagement if you put your best food forward by helping newcomers understand what this thing is. But hey, maybe I'm wrong.

(Not my submission)

I'm biased, because I already know what tinygrad is, but there's a link to the main github page for the repo at the top of the page. There's "get higher engagement" but there's also "readers here aren't drooling morons and know what a github is and can click on the link to the repo and find the README.md". But hey, maybe I'm wrong.

Is "readers here [ARE] drooling morons and [DON'T] know what a github is and [CAN'T] click on the link to the repo and find the README.MD" the only other alternative? There's no middle ground? There's no "readers here aren't drooling morons and know what a github is and can click the link to the repo and find the README.md, but they're also busy and their attention is limited and so some of them won't be bothered to take those extra steps to learn about your project if you can't be bothered to put your best foot forward, and it's not a moral failing on either side, it's just the way it is"?

fair.

>experimental backends for not requiring any userspace components like ROCm or CUDA.

That's pretty wild

This is a tiny autograd library alternative in D language:

https://code.dlang.org/packages/tiny-autodiff

Unlike other autograd libraries it utilized native D Mir GLAS linear algebra library [1]:

[1] Numeric age for D: Mir GLAS is faster than OpenBLAS and Eigen

http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...

What's the point of the 8000 LOC limit? Has anyone worked in a project with a LOC limit? Why was the limit in place?

It's just a way to keep the code size in check, make sure it can be read and understood relatively easily. Don't overthink it. I doubt much, if any, research went into picking the limit. The line width is over 120 in many places, and the code inevitably ends up looking like

  cache_key = (device, st, dtype, op, arg, tuple(ref(x) for x in srcs)) if base is None else (st, ref(base))

Seems the code sample contradicts your first statement

Indeed, I was making a point.

I think this might be an example of https://en.m.wikipedia.org/wiki/Goodhart's_law

The line count probably does still act as a limit on complexity overall but perhaps less than hoped for.

This is truly depressing because the aspirations of tinygrad are so appealing in terms of being concise, effective and maintainable. Then, instead, they throw comprehensibility entirely out of the window.

To stay Tiny

To keep it "tiny". (IIRC geohot started it because he thought pytorch and others were bloated and a simple ml framework would be inherently better)

Right now there doesn't seem to be much point. IIRC they had a 1000 LOC limit on the core part of the code when the project was early.

The README no longer mentions the limit and it looks like they just raise it whenever needed. Three months ago it was bumped to 6500 LOC. One month ago it was bumped to 8000 lines.

A tech debt ceiling so to speak then. There might be some use to it. It's still inevitably increased, but only after debate, discussion, and a lot of time in-between really considering the form and impact of the code being entered to fit within the constraint

To compare, the PyTorch repo has ~400k lines of C, ~850k lines of C++ and more than 1.5 million lines of Python code.

PyTorch does more than tinygrad, but does it really do 343x more things?

Probably.

If PyTorch does the 1-2 things you need and Tinygrad doesn't do, then what are you going to use?

The Python source distribution has long maintained the philosophy of “batteries included” – having a rich and versatile standard library which is immediately available, without making the user download separate packages.

https://peps.python.org/pep-0206/

OTOH:

  Simple is better than complex.
  Complex is better than complicated.

https://peps.python.org/pep-0020/

PyTorch of course. Or alternatively a lib or custom code on top of TinyGrad. Is that a problem?

uh, ya? lol

Easily

geohot explained on one of this streams, and per my terrible memory: “tiny” is a way of expressing the architecture constraint that the system should not attempt to target [(many hardware architectures and their optimizations) * (many model, training, etc etc variants)] like PyTorch - which requires maintenance of a shit ton of code and a staff/community behind Meta. Instead, tinygrad should provide core abstractions that can be composed to accomplish a similar set of targets but for only one hardware architecture (for now I guess). He is releasing a companion hardware item which would fund the development I believe.

I think you massively underestimate the complexity of pytorch. Even if we exclude all GPUs except for AMD, and exclude clang (required for AOT engine), pytorch depends on almost every ROCm library. And inside it depends on original Triton library, and on forked Triton, and on aotriton, which depends on forked MLIR (because AMD MLIR don't contribute these changes to upstream), which depends on another forked LLVM/Clang (because LLVM api is not stable enough for them, I guess). And then there is MIOpen/rocBLAS/hipBLASlt/hipSOLVER/rocFFT/etc - libraries with gigabytes (!) of autogenerated code. Additionally, there are dozens of smaller linked libraries like oneDNN, LIBXSMM, magma, numpy, openBLAS, all needed for running "things". So even without autogenerated code, consider multiplying 1.5 million LOC to 100.

It used to be a 1,000. I guess it’s just a reminder to be succinct.

Cyclomatic complexity would be a better measurement.

Looking at the code base right now, apparently to produce some of the most unreadable code possible (https://github.com/tinygrad/tinygrad/blob/master/tinygrad/re...)

LOC limits have to be one of the worst incentives you can give programmers.

The only one I can think of the dwm window manager (https://dwm.suckless.org/), that used to prominently mention a SLOC limit of 2000. Doesn't seem to be mentioned in the landing page anymore, not sure if it's still in effect.

There are benefits of having a low number of lines of codes, e.g. if you want to print out on a paper (and reduce the number of pages), or store on a disk with a limited storage (although number of bytes is a more useful measure, then), or if you want to read it to understand it in less time than a longer program, etc. Of course, the limit of number of characters on each line, is also necessary, then.

However, that doesn't solve everything. Many things it does not accurately measure, e.g. complexity, number of stuff in one line, program speed, memory usage, etc. Those are other things to measure, and it can be helpful to reduce memory usage etc, but that is not the number of lines of codes.

No new features.

Reply on news.ycombinator.com