Hacker News Reader

Comment by 4bpp | Hacker News Reader

Plenty of claims about it, e.g. here as a "fact": https://github.com/ggerganov/llama.cpp/discussions/638#discu.... I don't think occasional expressions of lingering doubt (still couched among positive language like calling it a "miracle") can offset all the self-promotion that clearly seeks to maximise visibility of the implausible claim, even as it is attributed to others, as for example in https://twitter.com/JustineTunney/status/1641881145104297985... . A cereal manufacturer would probably be held responsible for package text like "Fruity Loops cured my cancer! - John, 52, Kalamazoo" too.

View on HN

Where's the 30B-in-6GB claim? ^FGB in your GH link finds [0] which is neither by jart nor by ggerganov but by another user who promptly gets told to look at [1] where Justine denies that claim.

  [0] https://github.com/antimatter15/alpaca.cpp/issues/182
  [1] https://news.ycombinator.com/item?id=35400066

These all postdate the discussions that I linked (from March 31st). By April 1st JT themselves seems to have stopped making/boosting the claim about low memory usage.

I used your link.

I don't read that as a claim of fact at all. From the link you shared:

>Now, since my change is so new, it's possible my theory is wrong and this is just a bug. I don't actually understand the inner workings of LLaMA 30B well enough to know why it's sparse.

I haven't followed her work closely, but based on the links you shared, she sounds like she's doing the opposite of self-promotion and making outrageous claims. She's sharing the fact that she's observed an improvement while also disclosing her doubts that it could be experimental error. That's how open-source development is supposed to work.

So, currently, I have seen several extreme claims of Justine that turned out to be true (cosmopolitan libc, ape, llamafile all work as advertised), so I have a higher regard for Justine than the average developer.

You've claimed that Justine makes unwarranted claims, but the evidence you've shared doesn't support that accusation, so I have a lower regard for your claims than the average HN user.

The very opening line says

> I'm glad you're happy with the fact that LLaMA 30B (a 20gb file) can be evaluated with only 4gb of memory usage!

The line you quoted occurs in a context where it is also implied that the low memory usage is a fact, and there might only be a bug insofar as that the model is being evaluated incorrectly. This is what is entailed by the assertion that it "is" sparse: that is, a big fraction of the parameters are not actually required to perform inference on the model.

I think you are making a lot of soup from very little meat. I read those links the same way mtlynch read them. I think you're looking for a perfection of phrasing that is much more suited to peer-reviewed academic papers than random tweets and GitHub comments taken from the middle of exploring something. Seeing your initial comment and knowing little about the situation, I was entirely prepared to share your skepticism. But at this point I'm much more skeptical of you.

Reply on news.ycombinator.com