Hacker News Reader

Comment by pmontra | Hacker News Reader

> Writing code with AI? I could swap to a local model. Costs nothing.

Costs nothing? I'd have to buy a desktop computer with a good GPU to run that model. How much would that cost?

You can get a second hand MacBook Air 16GB with Apple silicon from a couple of years ago for about $1,000 and get a decent performance from Llama 3.1

Right? Or a second hand desktop for $300 and put a $300 3060 in it and get even better performance. So many options.

I think the point is that anyone can enter the AI cloud market to compete with the fat cats if they get too pricy. NVIDIA's de-facto standards are the closest thing to a market moat. If AI ends up highly tied to NVIDIA's standards, then NVIDIA could become the Microsoft of AI. It's hard to know how tool chains will evolve to answer that though, it's new territory. (It could be all moot if the bots take over and eat us humans. I taste like chicken.)

You need to own that or buy someone else's compute to run any LLM. The marginal cost of the open local models is free.

You can get a LOT of compute credits, at the public rates the cloud providers charge (not OpenAI's discount) for the price of an NVIDIA 4xxx with 24GB or more of RAM. Enough to do what most people need AI to do for a long time.

To run a 7b model? Not much. To run the 405b model mentioned, at interactive speeds, at reasonable quantisation? A lot.

Reply on news.ycombinator.com