Hacker News Reader

Comment by rmason | Hacker News Reader

Just listened to Lex Fridman's three hour interview with George Hotz this weekend. He spoke about his new company, Tiny Corp.

Tiny Corp. will be producing the Tiny Box that lets you host your own LLM at home using TinyGrad software.

The tinybox

738 FP16 TFLOPS

144 GB GPU RAM

5.76 TB/s RAM bandwidth

30 GB/s model load bandwidth (big llama loads in around 4 seconds)

AMD EPYC CPU

1600W (one 120V outlet)

Runs 65B FP16 LLaMA out of the box (using tinygrad, subject to software development risks)

$15,000

Hardware startups are extremely difficult. But Hotz's other company Comma.ai is already profitable so it is possible. I find the guy extremely encouraging and he is always doing interesting stuff.

View on HN

Is $15,000 really an "at home" sort of price?

(If money is no object, why not grab an oxide.computer rack? Assuming you have three-phase power, of course...)

I don't think the $15k price tag is intended for regular consumers. I didn't listen to the Lex Fridman interview (my patience starts to wane after the first two hours), but I did listen to another of Hotz's interview on another podcast, and I believe he referred to this their "Tiny Box" as more of a dev box. It's for the engineers and tinkerers who want a capable machine for AI. So while expensive, it fills a niche for those who don't want to deal with linking multiple Nvidia GPUs together and deal with the power and cooling setup that come with that.

He's definitely thinking bigger than just dev boxes. Lately he's repeatedly mentioned building own chips to be somewhere in the future. How plausible that is is another matter

Right, that’s the long term plan. I’m speaking specifically about the $15k first release of the tiny box.

Right, especially when it's not clear if such a machine will be needed a year or two from now to run a model with the equivalent performance.

At this rate it may not be needed in a few months to run a model of equivalent performance.

At work I've been successfully running llama2 variants on Mac M2 systems with more than adequate throughput for small experiments and examples. Can also boot a 4x Nvidia T4 instance on AWS (g4dn.12xlarge) to larger models.

I feel you'd need to be doing inference on a 24/7 basis for at least a year to justify the price of this machine.

That's roughly the price of two original Macintosh computers adjusted for inflation.

Does oxide do GPU builds? I thought they're classic server racks (CPU)

> three-phase power,

Think for the tiny grad it's run off single, but for the US the GPUs are power limited given the 110V grid.

We (oxide) aren't doing a GPU focused product yet, that's right.

A new car, especially electric, costs significantly more than $15,000, yet people buy them and bring them home daily.

Transportation is a universal need, and the time savings over other forms are often worth the premium. Doubtful the same can be said for these.

> Is $15,000 really an "at home" sort of price?

For something that is... going to hallucinate? I don't get it?

I mean, people spend way more than that on random art paintings and sculptures right?

With this, it could always be repurposed if something new and shiny comes along

> With this, it could always be repurposed if something new and shiny comes along

I mean, the oxide.computer specs make me go "this would be the most killer gaming machine in all of existence" ... though it's a server rack meant to host an entire private cloud for an entire corporation, I still would love to glom together as much resources as can fit into a single VM for gaming purposes.

George Hotz misread the NVIDIA Inception pricing as per unit instead of as a rebate, believing the GPUs are 80% cheaper than they actually are.

Why buy this instead of building your own setup with X 4090s?

I think he mentions this in the podcast. One of the challenging problems they're attempting to tackle is enabling the build to work with standard 120 volt wall sockets. He mentions throttling the system (GPUs specifically) to use less power at cost to performance.

That and setting up the software to work w/ an array of GPUs rather than one seems difficult.

I don't know if he actually said this, but power limits on gpus do work. I've been limiting my gpu to about 60% max power during long AI workloads and I lost about 10% of performance. A 40% reduction in power usage seems worthwhile.

It probably depends on your gpu though.

Undervolting would probably be even better.

same here with the 3090 I use. Power limited to 60% and I barely notice a difference when experimenting around with AI or playing games. Really good to use in a SFF PC

What OS/software are you using to throttle the cards? I'm interested in doing this on Arch

    ~ > sudo nvidia-smi -pl 300
    Power limit for GPU 00000000:09:00.0 was set to 300.00 W from 450.00 W.

You got the Arch solution. For anyone else on windows: you can just use MSI Afterburner (regardless what you have an MSI card or not).

144GB Vram vs 48?

And once you buy 6 4090s for 6x24=144, and add the other components, you are not that far off from 15k. The 4090s are still a bit cheaper and with a more standard setup, but it's close enough to consider the alternative.

Good luck running that on 1600W and keeping it cool.

What podcast (or otherwise) was this?

Was it this one? https://www.youtube.com/watch?v=dNrTrx42DGQ

Yes, that's the podcast I watched. He also talks a lot about comma.ai which is now profitable on $5 million of VC funding. I think that he has raised a similar amount of funding for Tiny Corp.

Think of it like an ac unit, they are expensive but indispensable. I have always envisioned locally trained ai and now it is happening!

You probably need an AC unit too if you're going to be dumping 2000W of heat into your living space.

why are you valuing George Hotz (or Lex Fridman's) opinions in this at all?

Imagine how many 4090s you could buy and run in a cluster for $15,000 though

I hate to burst your bubble, but more than two 4090s is going to put you at BoM + labor costs around $15k. Especially if you have to upgrade your electrical and hvac.

They seem to have six, either 4090s or 3090s judging by the total amount of VRAM. How many more do you think you could get with $15k considering all the other hardware costs? I doubt you could make it more efficient at this price point.

It might be tough to make more efficient, but $15k seems exactly about the price of "stick 6 4090s in a decent box and throw in a couple grand for my troubles", versus any revolutionary hardware configuration. The way it advertises running fp16 Llama 70B feels a bit contrived too, given the prevelance of quantizing to 8 bit at minimum.

In my opinion the best hardware to run big models is to go and get a mac studio ultra. You have 192GB of unified RAM which can run pretty much every available model without losing performance. And it would cost you half that price.

> without losing performance

But isn't M2 ULTRA over 20x slower than this thing? ~30 TFlops vs 738.

By losing performance I meant you don’t need to quantize the model a lot since it fits in RAM. My bad for not clarifying it.

According to this [1] article (current top of hn) memory bandwidth is typically the limiting factor, so as long as your batch size isn't huge you probably aren't losing too much in performance.

https://finbarr.ca/how-is-llama-cpp-possible/

The Tinybox is planned to use AMD, not NVIDIA. The GPU they're using to build Tinybox is the 7900 XTX.

Reply on news.ycombinator.com