My solution is to see what four or five-year-old equipment I can buy that will let me run local LLMs. I may only get six or seven tokens per second out of an i7, but it's a start. And best of all, I can turn the machine off when I'm not using it.
IMO, Migrating to small-scale local LLMs would be a significant improvement over using data centers.
LLM serving is most efficient when you batch a lot of parallel requests together. Data center solutions also have the advantage of collecting queries from around the globe, so the hardware can be utilized around the clock.
Having everyone serve their own local LLMs would produce a lot more memory demand. Not less. The same memory would be idle most of the time, and when it was used it would be used for 1 person instead of a batch of requests.
There are other reasons to run local LLMs, but solving hardware demand problems is not one of them.
It occurred to me much later that local machines could become a shared resource of a small co-op. Or it only runs off solar or energy harvested at times of excess generation (too much daytime solar, lots of wind, or overnight when demand naturally drops).
Exploring how to partition inference across many machines and shed load for idle sections of hardware would be a problem that could be solved by a community of smart people like us here
It's also resilience. I can still do AI stuff if the internet goes down.
This shift is probably inevitable, but it will vary significantly by region depending on prices for electricity. Look at, for example, the difference in fundamental homelab build recommendations between Germans and just about anyone else. Electricity prices in Germany are so high that even a now expensive Raspberry Pi or other ARM board is often preferred over Intel/AMD builds due to low power draw (especially low idle power draw), an effect that adds up for a machine running all the time over years.
With local LLMs and the GPUs to run it, especially if you want a model available to you all the time and can remote into your local network to use it whenever you want, there's no escaping much higher power draws, even at idle. Wherever electricity is expensive, the electric bill can be a prohibitive barrier.
Eg: I shaved ~40W off the idle load on a server (250->210W) by doung nothing more than removing the redundant supply
A college buddy used to work at Motorola (I'm naming the company because they wouldn't mind this story being shared) back in the late 90's or early 2000's. They had redundant power to their campus, bought from two different companies, coming in on opposite sides of the campus, so that even if some backhoe operator cut a ground-based power line somewhere, they wouldn't lose power.
And yet, one morning, the power went off all across their campus. After a little investigation, they sent pretty much all their employees home at noon and told them "take the afternoon off, don't come back until tomorrow, you wouldn't be able to do any work anyway". Turns out that although the power lines came in at opposite sides of their campus, somewhere a few miles away both of the power lines feeding their campus ended up running through the same underground conduit. And yes, a backhoe had managed to cut that conduit and break both of the lines they depended on at the same time. They had a single, VERY non-obvious, point of failure, and the backhoe had unerringly homed in on that SPoF.
I'll second this. The combination of Whisper + LLM makes speech recognition fantastic. I occasionally have arm pain from typing, and this is a Godsend.
I don't use it to write code - but in my experience stuff like emails + docs was the greater source of pain (one generally types slower while coding).
And hand problems was an aggregate thing. first chronic pain then fine motor control and tremors. The pain flares up again if I try to use my hands too much.
I don't see the llms have me trapped like a tethered elephant anymore than the other layers of entrapment that we live with. I would argue entrapment started when we learned the secret was banging rocks together.
Or did you mean some other kind of entrapment?
And yes, running local ones actually are the best of both worlds I think, because it puts less pressure on the memory/storage needs as a whole.
> how do I do this one piece of a query in MySQL, since I'm not a MySQL DBA
Honestly for most things, the actual documentation is the best resource. I needed to learn to stop looking for solutions to my issue on various fora (like SO) and just read the primary documentation from the vendor. Have a look at dev.mysql.com or mariadb.com/docs/server
In that example specifically, it was a bit of "let me see how this works and how good it is" too.
Microsoft has a great racket going on there. Vibecode shitty, unreliable services, offer wrong information in the slop documentation, then charge an arm and a leg for premium support which, in many cases, will just be an employee regurgitating more wrong AI slop information to you.