Hacker News Reader

Comment by robertritz | Hacker News Reader

MLX only works for fp16 right now. If it ever works quantized I will almost certainly move my app over to MLX instead of llama.cpp.

My app also uses a very small (30MB) PyTorch model and shipping it requires an extra 100MB for PyTorch in the app. Very very stupid.

I think its important to remember that last mile inference is still pretty bespoke for most things. If we want to see gen AI stuff take off and now have the big cloud providers in charge this needs to be fixed.

Apple is in a good place to solve at least part of the equation.

View on HN

Apple will most likely introduce their own API for an LLM-powered Siri which devs can use in their apps. This way, Apple keeps their control and can fully optimize the SiriGPT (or whatever you call it). Very unlikely that Apple would just provide the hardware, good amount of RAM, a framework competing with Pytorch, all out of the goodness of their heart. I use Apple products but I know they're way past the point of doing things just for the sake of fcking doing something awesome.

Becoming the AI dev and inference platform of choice and getting that nvidia profit margin is probably plenty enough of a reason. Right now it's CUDA on linux PCs.

I don't see Apple taking NVidia's crown for training. Apple's hardware acceleration is fine for low-power on-device inference, which they were designed for, but competitive training requires an order or three magnitude more power, which they aren't.

I keep a bookmark of your comment for my 2030 me. Just to check whether that prediction will realize: Apple will ALSO storm the LLM market!

Unless you also predict that Apple will release datacenter systems a-la Grace and Instict, I don't think they're even in the runnings. AMD is only competitive in the LLM market because they sell extremely cheap and fast compute hardware at the same scale Nvidia does. As of today, Apple doesn't sell any hardware that can go toe-to-toe with a DGX system. They also have a lot of software problems (VM limitations, poor GPU API support, limited integration with open-source, etc.) that would need to be fixed for parity with Nvidia or AMD.

Apple will definitely push for on-device AI, but even in 2030 I firmly believe that they won't be leading the industry in performance. I'd be surprised if they even supported anything other than their proprietary CoreML by then.

Ditto on this. I want to not buy an A100 for $20k, or even consumer GPUs, but the truth is that for LLM inference, to run large models like LLaMa2 70b with INT4 quantization so it could fit

A100: 1248 TOPS

MI250: 362.1 TOPS

M3 Max: 18 TOPS

Yes, 18. Unless Apple has accelerated INT4 workloads but just forgot to document it.

Honestly, I’m an Apple fan, but when they go on stage and say “AI” they mean it can do speech recognition or tell a dog apart from a cat, or autofocus a camera. It can’t run ChatGPT-like things by a loooong mile.

all they care about is the value of a walled garden.

I just went through all the commercial options for local LLM hosting and they're definitely poised correctly because they have the correct amount of memory in their machines.

they're not going to have a come to Jesus moment.

If the model could be made to work with llama.cpp, then https://github.com/abetlen/llama-cpp-python might be more compact. llama.cpp only supports a limited list of model types though.

Use llamafile

Reply on news.ycombinator.com