Petaflops to the People: From Personal Compute Cluster to Person of Compute
latent.space
latent.space
we were really honored that George granted us the first interview on tiny corp. We even set up a dedicated podcast youtube for the first time (Alessio and I are relative youtube newbs) and George was kind enough to come back to Twitter just to help us kick it off: https://twitter.com/realGeorgeHotz/status/167120786704580608...
Personal highlights for the HN crowd:
- the three theses of tiny are all interesting: the CISC/RISC analogy, why you should start with software for commodity GPUs rather than taping out chip design, why to avoid Turing complete layers in the ML stack
- I loved comparing his natural debugging and CI style to John Carmack - developers who assume geohot is a cowboy coder might be surprised at how seriously he takes developer experience
- Dishing on tinygrad vs ggml and Mojo (i'll just leave it there, its good)
- how everyone is in this mad quantization rush but nobody's putting up benchmarks to show that it works (tinybox is resolutely supporting non quantized LLaMA)
- hardware design constraint choices for tinybox (<60 decibels, single power plug, etc)
and then we of course indulged in the array of offtopic convos for fan service:
- GPT4 is just a mixture of experts - 8 GPT's in a trenchcoat
- Elon vs e/acc
- Rich Sutton's Bitter Lesson
- The Goddess of Everything Else (go listen to him read it https://mixtape.swyx.io/episodes/the-goddess-of-everything-e...)
and of course, he's doing all this so that you'll consider working on/for tiny. job application details on https://tinygrad.org/
- Idk how to say this but surprisingly not arrogant. we both prepped by watching his lex fridman and prior livestreams, so its not like we were dealing with an unknown quantity, but I've never seen him interact with "normal" (technical, but not his level of technical) people like us before. he was pretty patient I think, except I definitely got roasted for suggesting you could jailbreak the tinybox. instantly regretted it but he only joked once aboout it which was super cool :)
- quite reasonable expectations for his AMD frustrations. the fact that Lisa Su got personally involved and they listened was also fantastic. this isn't a "tech diva flipping the table because they stubbed their toe" story - AMD was found wanting in basic documentation requirements, he raised the issue, they took a while but eventually fixed it. Great partnership all around imo
- hiring in the age of AI is really going to get strange. he thinks we're ~20 years out from developer jobs being gone. Yes he is essentially asking people to do work for small bounties. in some circles, that's a no go. but it basically guarantees that the candidate will be a fit, and is fine if you want to keep the team tiny anyway. he also changed his mind on remote work which is cool.
- this was the first time i heard anything materially different about GPT4 architecture being a mixture of experts rather than a 1 trillion super model. does raise the question about what GPT5 will be built on if we have indeed run into the limits of scaling model size.
- notes to future interviewers: george is ultra open (he identifies strongly as a streamer, is very comfortable with what he is saying and doesnt need a ton of prep or post recording review) - just be super attentive to what he does and does not want to talk about and let him riff and he will indulge. i think we were maybe like a 7 out of 10 on getting the best out of him, but hopefully we will be a lot better by the time we go down to San Diego for the tinybox launch :fingerscrossed:
- He gave us advice on whether or not we wanted to go big and be the next Lex Fridman or stay niche and I think we were pretty convicted on wanting to stay niche and technical. To my knowledge there's no other podcast in the world that could have had this deep-but-still-kinda-accessible conversation with George and I strongly feel like something like this has to exist.
I don't think this is true. llama.cpp has historically been very conscientious about benchmarking perplexity. Here's a detailed chart of baseline FP16 vs the new k-quants: https://github.com/ggerganov/llama.cpp/pull/1684
While most evals aren't currently evaluating performance between quantized models, there are two evals that are:
* Gotzmann LLM Score: https://docs.google.com/spreadsheets/d/1ikqqIaptv2P4_15Ytzro...
* llm-jeopardy: https://github.com/aigoopy/llm-jeopardy - You can see that the same Airoboros 65B model goes from a score of 81.62% to 80.00% going from an 8_0 to 5_1 quant, and 5_1 solidly beats out the 33B 8_0, as expected.
Also, GPTQ, SPQR, AWQ, SqueezeLLM all have arXiv papers and every single team is running their own perplexity tests.
Now, that being said, every code base seems to be calculating perplexity slightly differently. I recently have been working on trying to decode them all for apples-to-apples comparisons between implementations.
Fundamentally, I'm not sure if perplexity on a dataset that's pretty much leaked everywhere will properly measure the task performance of LLMs. But open to being wrong on this.
Losses appear to be in-line (possibly less) than increases in perplexity.
My intuition is that the dataset you use for testing perplexity (c4, wikitext2, wikitext103, whatever) shouldn't matter so much as long as its a representative sample of data it's read. You're calculating cross entropy loss, so you want to have data you know it's read to see how much each quantization gets more wrong.
And thanks for the https://pypi.org/project/lm-eval/ pointer, have to get this in tinygrad.
I favor a radically stupid approach... just make a huge array of 4 bit in, 4 bit out, LUTs in a cartesian grid, clocked 2 to phases like the colors of a chess board. This results in cells that are NOT Turing complete, but an array that clearly is.
The delays going across this "bitgrid" are huge, but each and every cell is computing, unlike any existing system. You can trivially map things, you can route around bad cells, flip the programs in any direction, actually wall off bits to ensure only certain I/O for a given area of computation.