For most of the world, open-source AI is the only way forward
techstrong.ai
techstrong.ai
This doesn't read as revolutionary, it reads as accepting a premise without explaining how and why it will and must be the case.
Some of my favorite parts of the internet - wikis and forums and blogs - are special to me because there is no procedurally generated mediator between me and the information. I don't want a summary that was never intended by the author. I don't want the rough edges shaved off every blog until they all sound like they were written with an identical voice. Maybe some people would like that some of the time, but all?
They(we) don't consume directly from the firehose of the internet, but from whatever "The Algorithm" has curated for them(us).
It's the same way the algorithm telling us what to see when we just want to see the raw feed.
In all cases, we could and can have both.
> It was only in 2025, as memory prices began an unprecedented surge, that the memory makers started to build new fabs targeted at HBM, all slated to start producing chips in 2027 or 2028.
If you want to argue that this is different from all previous RAM shortages, you can, but the burden of proof is on you to show the difference.
this time demand doesn't stop. there is an exponential demand for tokens.
[citation needed]
There is certainly economic pressure to create an exponential demand for tokens, but we've already seen a pullback from the costly "token maxing" companies were pushing last year.
It's also pretty unclear to what degree the RAM shortage is driven by inference (versus by training). We're rapidly approaching the point where frontier models are "good enough" for everyday use, are at some point we're going to hit diminishing returns on training new trillion-parameter models...
Possibly it's the same price range, allowing for inflation.
So yeah, I think models on local hardware will be quite common soon among the tech savvy (such as people creating software).
E.g. grok isn't truly multi-modal, it has a callable tool that is a separate VLM it invokes on image URLs or files (for a long time it was grok-1.5v, but I think they have upgraded now, it was pretty bad).
And then you have the small summarizer models for the CoT/thought traces, the guidable summarizer models for the standard browse tools, etc.
There's a ton of stuff that can use an aging GPU.
A100 -> H100 was >3x tokens per joule, H100 -> B200 >10x. There are significant low-hanging fruit still available in architectural efficiency, and the vendors are chasing them.
This is the big risk for AI companies that I feel is not being sufficiently priced in. Almost none of the investments they are making are durable, the depreciation schedules for everything but the real estate should be less than 24 months. Until the hardware is stable enough that you only get double-digit % improvements per generation, it should almost be counted as opex.
As it stands there's way more demand than supply. The new GPUs are going to run frontier models while the older ones serve smaller ones.
That said some of these are running in tents hooked up to mobile turbines. I can see some of those going away but generally I think you'll see them used until they start to fail in 5-10 years.
I do hope you're right that it will get cheaper over time (it should), but right now 32GB of VRAM is not affordable to a lot of people. You're talking ~$4500 just for the GPU, or $800 ish used if you can find one.
It's a tad less efficient and a bit more of a hassle, but still a good experience for only a fraction of the price.
I imagine having multiple providers competing will drive down hosted versions of open weight models drastically.
I was thinking more of the providers of inference on open weight models that openrouter proxies to.
And we've barely started to scratch the surface on helping open-weight models "be the best they can be", with cloud burst parallel sampling and prompt mutation. Looking for best probabilistic results for a prompt, and looking for best prompt variants for a task. Adaptively scaling computes at generation, not just training.
And speculatively, if agentic coding is naturally a multiplicity, what UX might enable human devs to dance with that quantum superposition? Rather than quickly collapsing to one monkey and its keyboard.
https://huggingface.co/Jackrong/Qwen3.5-27B-Claude-4.6-Opus-...
My use of the term model here is confusing
Gotta remember inflation here.
$1K in 1995 was roughly equivalent to $2K now and wouldn't have been a particularly "good" machine then.
In 1982 the Commodore 64 started at about $600 bucks, also roughly around $2K today.
If you outgrew that, beefier machines back then were A LOT. It was easy to find $2k+ towers and (especially) laptops even into the 2000s, and a lot of those would be $5K+ equivalent today.
Certainly the transistors/chip or transistors/$ or flops/$ have not been progressing at the same exponential rate as during 1970-2010. There is still progress, but it's rather slower.
As you point out it's really cost per transistor or cost per flop that we mostly care about. I'm finding it hard to find a succinct and clear plot, but I believe one is provided by Our World In Data on "GPU computational performance per dollar" [1] which, to my eyes, clearly shows exponential growth in computational power per dollar.
The picture for storage is a little more muddied but if you squint just right you might still be able to recover an uninterrupted exponential growth [2].
In my view, it's pretty clear that advances in AI have progressed so quickly because GPUs have been keeping up with the exponential growth of computational power (per unit cost).
Exponential growth in this area is usually characterized by "S-curves", where one technology gets saturated but the exponential increase in power or decrease in cost is picked up by another, adjacent, technology, that allows the growth to continue. For compute it's CPUs to GPUs. For storage it's platter drives that are now being overtaken by SSDs.
The more general phenomena is called Wright's law, or experience curve effects [3].
[0] https://en.wikipedia.org/wiki/Moore%27s_law
[1] https://ourworldindata.org/grapher/gpu-price-performance?ySc...
[2] https://ourworldindata.org/grapher/historical-cost-of-comput...
Especially because the world is likely to persist, at least for a while, in state where computing hardware demand drastically exceeds supply resulting in high prices for hardware. So why wouldn't you want to max out utilisation and amortize costs, at least for typical (non sensitive) use cases.
Started with computers around 2009 and later bought an oldish computer (a pentium 4 PC) for the equivalent of 50 usd. Codeblocks and Python Idle were free at the time (C and Python were the first languages I learned). The barrier to programming has always been low as the only thing you needed was books (the internet made things easier) and access to a PC (I had friends with laptop and my school lab).
It's definitely worth investing in self-hosting the agent infrastructure around the model though: all the documents, knowledge base, all the connectors, the agent itself to run on your hardware
By all accounts, all AI companies starting with open are doing proprietary stuff. All models delivered for free as "open-models" are just freeware as no source is really provided.
Collaboratively trained open-weight models, is my understanding.
https://webtv.un.org/en/asset/k14/k14ej1ucqu?kalturaStartTim...
(if that link doesn't work, it starts about 12 minutes into the start)
Great analogy to the fear of the printing press being really bad news in that it enabled the rabble to get aroused.
All that's needed is another sovereign debt crisis to spark what is essentially dry tinder and I think the EU is a lot closer to collapsing than anyone even remotely realizes.
I have been hearing that will happen next year for the last 20 years. No this time it isn't different.
The European “problem” is they live to have a life whereas in the USA we seem to live to work.
> the investors who backed these foundational model companies who will hold the bag
Is awfully bold to assume that private credit is who will be holding the bag here. The IPOs are coming to shift the risk to the index funds & retail. Once insider lock up periods expire, I suspect a massive sell off.
> On the one hand you have—the point you’re making Woz—is that information sort of wants to be expensive because it is so valuable—the right information in the right place just changes your life. On the other hand, information almost wants to be free because the costs of getting it out is getting lower and lower all of the time. So you have these two things fighting against each other.
Information may want to be free, but the humans creating it still need to eat and pay rent. Copyright isn't necessarily unethical more than its a flawed tool, and lasts far too long in the law's current state. It needs to last only and exactly as long for the original creator to profit from the work for a specific duration of time, and then thats it.
It is only because of rampant greed and capitalism that information is not free. There is nothing inherent about the collective knowledge of mankind that lends itself to being proprietary and expensive. Otherwise human society literally could not have evolved.
I'd struggle to find an idea, art, technique etc... that wasn't an extension of something that came before it.
edit: come to think about it I think the ratio of one drop to one bucket is vastly over estimating the ratio of the trainer's effort.
Preventing a handful of massive companies from continuing to be the only ones able to make money off that, not only unimpeded but with overt or covert state assistance (regulatory capture, ownership, whatever), at least puts an end to the worst of the abuse.
If we have broken the idea of copyright, and we do indeed appear to have broken the idea of copyright, why should trillion dollar companies owned and controlled by strange or psychopathic weirdos and their circle of investors be the only ones benefiting? Why do Sam and Dario or the US government get to decide when and for whom the tap is turned on?
Otherwise, everything you do belongs to all of us, because you learned on total intellectual and artistic output of human kind too.
I predict that mech interp and things like Neuronpedia will matter more and more over time, and the frontier providers are disincentivized from providing those tools
what has Linux won? Servers? sure
The bitter lesson would claim that we only need one model with more data thrown at it, but I’m a bit skeptical that we’ll end up with “the one true way” of building a model, and I think there will be model tradeoffs that we can pick from as the industry matures.
We need large scale open weights models just as capable as what's at the frontier.
And we need the ability to rent compute and spin up the weights easily. One-click, easy enough for anyone. Easier than nerd tools like ComfyUI, Claw, and node graph garbage.
Freedom is owning very large scale weights. Anything less is subsistence.
This is the wrong approach that will turn us into serfs. We need big honking models that do what the leading foundation hyperscaler models do to within a few percentage points of measured performance.
The small-scale models are not productive, and the duct tape solutions built on top of them are hobbyist-tier "year of Linux on desktop" toys.
I imagine fedora-wearing, crypto-shilling, coupon-cutting boffins every time I see small weights thing lauded as the future. This is the Pine Phone F-Droid of AI.
"SMS works most of the time on my phone, I swear! I don't really need my banking app!"
That is not big model energy.
Nothing outside of the top ten is worth spending any time on, and we need to focus on models that bridge the gap.
You're talking about impractical toys for highly technical people wasting their own time. That doesn't move the needle or have any economic impact on the competitive landscape.
We need sharp teeth that bite at the legs of the top-tier foundation labs and hold them back from running away with the prize.
We've been through this time and time again over the last thirty years. It's the same shaped problem as before. We don't need toys - we need real infra for real people paying money to do work. Not freeware for freeloaders who don't spend and invest in the problem space.
Large models fit that precisely, because it forces investment into a wide variety of open infra, routers, inference engines, etc. Not to mention the weights ecosystem itself.
We need the right tool for the job. Certain models have minimum energy expense no matter what the task is and that's often wasted, both on the scale of some tasks and also repetition.
There is a place and a need for large models, local models, and single purpose models. The same way there is a need for HPC and single board.
If you tell the open weights labs that you want tiny models, you'll get more shitty tiny models.
If tell them you don't want tiny models and you want big models, they'll change their focus.
> We need the right tool for the job.
The world has enough shitty tiny models. You can frankly distill your own without wasting the time of foundation labs.
> energy expense
You are so far removed from the discourse that I think I'm wasting my time talking to you. This is not about energy in the slightest.
> There is a place and a need for large models, local models, and single purpose models.
This is where 99.999% of the demand is. And if you look, you'll notice that most of the token usage is accruing to hyperscalers because the market isn't meeting this demand with enough alternatives.
The robotics industry will produce energy efficient models. Let them be the ones to press for that. Stop demanding cheapo LLMs, image, and video models. Big is all that matters in this domain.
My hunch is that the energy/water usage of the data centers is a whole lot more efficient than everyone running at home, but I'd be interested in seeing real data on that.
I'm looking forward to this too. It'll be incredibly useful.
On the energy front, I assume less efficient, but I also think there is a tradeoff in efficiency versus freedom, that's why I have my own hardware.
Once we get to fully renewable (as a country), there should be no water involved in electricity generation as well.
So: if you're running the models on your own machine, presumably you're not running them as often, and air cooling is sufficient. But, at the same time, this is less efficient in terms of hardware use; the data centers need water cooling specifically because they're getting more bang from their buck from their hardware, by running their hardware harder.
So that's the tradeoff: more hardware-use efficiency means more water usage.
> [Google’s] largest data center, located in Council Bluffs, IA, withdrew an average of 3.9 million gallons of water and consumed 2.8 million gallons per day.
https://mostpolicyinitiative.org/science-note/data-center-wa...
You can start adjusting your consumption of those products.
> distributed LLM inference. We are at a point where no single person can setup a rig to run a SOTA model, it is just too expensive. So we must build and adopt frameworks that allow individuals to share resources to run SOTA models in a distributed manner. That way they will also be non-censorable by governments.
Also The only way to prevent that one entity weaponizes it, is by giving EVERYONE access to it.
You rent your fiber optic internet. You're doing just fine. The world isn't collapsing because you don't own the hardware racks, routers, and fiber lines.
This crazy zany P2P communal infra is Arch Linux coded - too much work, aimed at the 0.001% of users, and the juice isn't worth the squeeze.
It sounds the exact same as being mad at your ISP, so you want to build a mesh internet protocol over microwave dishes and share with everyone in your neighborhood. That was a thing in the 00's. And it predictably got nowhere and delivered no value.
Nobody's got time for this kind of stuff except for hobbyists. It's not solving a real problem. Nor does it attract business investment to grow into a healthy offering.
Just build OpenRunPod instead.
The problem is that the world doesn't have access to frontier open weights. That's the only problem to focus on and solve.
The quickest way to get there is to use large scale open weights instead of little rinky dink RTX distillations. And the easiest way to run them is by renting H100s or using a managed service offering. There's nothing wrong with either of those options.
The infrastructure is the easy part anyway.
Even if it's too expensive to run the models on your own personal hardware, open weights may still make it possible to take power back from the big private corporations.