Hacker News Reader

Hacker News

Comment by threeseed | Hacker News Reader

Full thread

threeseed

> For instance, serving a single 175 billion LLM requires at least 350GB of GPU memory using specialized infrastructure

Apple ships the Mac Studio which support up to 144GB of usable GPU memory.

Would be amusing if they were to release a Mac Pro with 300+ GB and dominate the LLM serving space.

View on HN

> Would be amusing if they were to release a Mac Pro with 300+ GB and dominate the LLM serving space

Is there any framework that can batch LLMs on Metal? I don't think GGML or MLC have it yet.

Otherwise that is just another reason they wouldn't be good for LLM hosting at this moment.

Anyway, the real disruptor is Intel. They could theoretically barge in with a 2x48GB Arc card and undercut the market that AMD/Nvidia refuse to dive into because of their Pro card clients.

It's coming to llama.cpp soon.

https://github.com/ggerganov/llama.cpp/pull/3228

Amazing!

Its really a shame that Apple (and AMD/Intel or pretty much any other infrence vendor) are not directly contributing to llama.cpp. The feature set is amazing and growing at a stunning pace.

I imagine the legal concerns around it are part of the reason. Where I am, our legal team has been very direct, that anything llama based cannot touch a work computer.

Curious to know why this is the case? I haven't played around with LLAMA yet but I figured it being open-source would make it more trustworthy than models provided by OpenAI.

> it being open-source

Absolutely not. The LLAMA license [1] is clear that it's not open source. It's for non-commercial, research only, and only by explicit permission from Meta. The weights were leaked, on 4chan [2], illegally, according to the license. Very very few people are using it legally. This interpretation is clear from its wording, and also matches the interpretation of our flock of lawyers.

[1] https://github.com/facebookresearch/llama/issues/266

[2] https://levelup.gitconnected.com/metas-chatgpt-is-now-illega...

Llama 2 doesn't use that license and llama.cpp isn't limited to using those models anyway.

True. That gives these companies a little less than two months to start contributing, with the easier license restrictions in mind.

But, Llama 2 is not open source [1].

[1] https://blog.opensource.org/metas-llama-2-license-is-not-ope...

I wonder what legal teams (such as y'alls legal dept) think of the new license.

You're correct, I confused it with LLAMA being more business-friendly (at least in theory) than some other models out there.

Apple's hardware advantage is something I'm hoping to see really unleashed with the M3 generation. The fact the A17 Pro has ray tracing support is giving me hope they can catch up with the incumbents quickly. It's honestly the one thing that has kept me away from their newer hardware-- I primarily use my computer at a desk and PC hardware (GPU primarily) is leagues ahead of what their very best can do. I honestly can't justify spending nearly 4k when Linux works great for everything I do professionally AND when I'm done with work I can still play a game.

3 most important things for Apple are:

- HW design

- UI

- UX

That Macs come with awesome VRAM is only because #3. Apple has no explicit incentive to make LLM-ready machines.

I disagree. ML is eating the world by storm and one thing Apple loves is high margins. Folks like myself would gladly pay that margin to someone besides Nvidia. I paid $2k for my last GPU and was happy to do it. That doesn’t even factor the fact that businesses would pay even more when the alternative are machines that start in the tens of thousands for just the GPU ability. GPUs right now are carrying fat margins and that makes more than enough incentive for Apple IMHO.

But you wouldn't be able to upgrade an Apple Mac when the next generation of LLMs is out.

You do get some money back for a trade-in depending on how old the machine is.

You can find many leaked email threads from Apple where they talk about the value of high powered local compute. It is a fundamental part of their strategy. They do not want the cloud approach for high end compute like Google, they want you to have high end and powerful devices in your hands.

Got a link to those emails?

I am wondering who will be the first to raise RAM capacity of their hardware offerings dramatically to win the LLM crowd. Seems like the way to win market share.

Agreed. I would love to have something with the FLOPs of a 3070/4060 and 80 gigs of even slower VRAM (not necessarily HBM/GDDR6x) that can run the XXB models.

That number is not quantized either. Quantize 175B param at 4 bits then it'd fit in ~120GB VRAM (one can fit 34B models at 4 bit quantization in a single RTX3090 24GB VRAM)

Reply on news.ycombinator.com