Comment by behnamoh | Hacker News Reader

Comment by behnamoh | Hacker News Reader

Parent Full thread

It seems after 1-2 years that the true power of LLMs is in DevOps. I got pretty excited when I tried GPT-3 (completion model), but as time went by and OpenAI shifted to chat models, we lost control over the LLM part and found new meaning in taking whatever model OpenAI made available as a blackbox and "chained" it to other tools we already had, like data bases, APIs, function calls/tools, etc. I'd say DevOps is exactly where open source is seriously behind; there are decent open source models but it costs so much to self host them, despite the full power and control we have on them (via text generation webui and the like).

OpenAI is playing the DevOps game (starting maybe with introduction of ChatML). Open source community plays the LLM and benchmarks game. Ironically, the two are converging, meaning that OpenAI's models are getting dumber (not the API) thanks to censorship and RLHF, to the point that open source models are even better than some OpenAI models in some aspects. On the other hand, open source models are getting better tooling and DevOps thanks to oobabooga, llama.cpp, etc.

I'm seriously waiting for competitors to change nVidia's monopoly in this space. Maybe Apple?

I think currently M2 max is best bang for buck running interface in open source model. But use case is so niche that Apple probably doesn't actively start supporting open source models. In the long run I hope some smaller company gets shit together and starts competing with NVIDIA.

The GPU support in ML frameworks however is really not impressive. I have a Macbook with M1 Max 64G RAM, I can load a 7b model for fine-tuning (Huggingface Trainer, Pytorch, MPS), but the speed is just too slow, can only reach to 50% the speed of an i5-12500 CPU in my tests.

At $6,000, how is M2 Max the best bang for the buck?!

One could get two used 3090s and setup a decent PC at lower prices.

Two 3090s don’t have 96GB of VRAM

That's true. But OTOH, one can't easily upgrade Macs.

With 96gb of VRAM, will you really need to? Personally I think apple servers might be adopted for AI and LLM workloads soon.

M2 max studio is 2000$.

> I'm seriously waiting for competitors to change nVidia's monopoly in this space. Maybe Apple?

I would have thought AMD is the obvious contender. They are #2 in GPU's, they have formidable programming talent (based on their advances with Ryzen vs Intel) and they have targeted AI as their goal.

Am I missing something?

AMD have repeatedly dropped the ball when it comes to software support for compute and AI. Their hardware is quite capable, but very few people can actually make it work, which means most of the existing models have poor AMD support.

This is getting better with ROCm and such, but that's Linux-only and only works for a subset of tasks.

Both Intel and Apple have better "out of the box" support for ML and the ability to invest more into making these things work (e.g. Apple have implemented Stable Diffusion against Core ML themselves)

Reply on news.ycombinator.com