Hacker News Reader

Comment by whoisjuan | Hacker News Reader

I like the idea behind this because large AI seems to be highly constrained by co-located computation and the costs associated with it (GPUs and energy).

There are many delivery and cost advantages to running a massive LLM in a distributed P2P fashion.

Weirdly enough, I see this as a real "web 3" opportunity. Corporations running large LLMs could run their models on a decentralized network and pay participants for their contributed computing capacity.

AI most significant headwinds are cost and the pace at which GPU capacity is being built. This seems like a good model to tackle both issues.

View on HN

> Weirdly enough, I see this as a real "web 3" opportunity. Corporations running large LLMs could run their models on a decentralized network and pay participants for their contributed computing capacity.

The same problem we saw with "web3" is here. If I were a "miner" in this case, why would I not go commercial-scale to gain efficiencies here. I could just build a real datacenter, and offer real contracts to real companies instead. It'd be cheaper for everyone.

Unless the expectation is that we literally can't get enough GPUs for all the datacenters, and we rely on the aggregate of consumers' integrated GPUs in their laptops? I think we'd just see companies not using LLMs before they got desperate enough to pay rando's for LLM processing.

If we compare this to crypto mining, most mining is done by big players with datacenters.

But it's still decentralized, and decentralization drives competition in a way that traditional B2B contracts cannot. The fact that anyone on the planet who can afford a GPU or an ASIC can be a competitor is significant.

For example, an RX 6800 will generate ~$0.34 per day minus electricity costs if you mine with it. That's the true value of that card on a global decentralized market. But renting a similar cloud GPU will cost about $0.30 per hour. 95% of that cost could be eliminated with a decentralized market.

> The fact that anyone on the planet who can afford a GPU or an ASIC can be a competitor is significant.

Except you can’t really make money. You need a data center to move the needle. If I was a company, I wouldn’t want any of my compute running in some kids dorm room or the basement of some house in the burbs.

> For example, an RX 6800 will generate ~$0.34 per day minus electricity costs if you mine with it. That's the true value of that card on a global decentralized market. But renting a similar cloud GPU will cost about $0.30 per hour. 95% of that cost could be eliminated with a decentralized market.

What about maintenance and redundancy? What if you need 2 for 12 hours and 0 for 12 hours? The value of cloud compute is not the rental cost of hardware (or mining cost?) it’s everything else. It’s scale, and maintenance, and geographic distribution, etc. it’s the nice GUI and support staff, it’s the SLAs and SDKs, etc.

Try renting a Mac on Aws - where a month will probably cost the same as buying it and consider why people may use it. Consider why there isn’t a decentralized marketplace of MacOS VMs despite this.

The average computer is not realistically capable of running LLMs effectively (because VRAM or RAM does not fit the full model).

You simply reward based on performance

“Run large language models like BLOOM-176B collaboratively — you load a small part of the model, then team up with people serving the other parts to run inference or fine-tuning.”

According to this excerpt, a node in the network doesn’t need load the entire model. Only a part.

It's a pretty naive idea (web3). Impossible to implement.

Care to explain why?

How do you calculate computing capacity? What is the output - AI gibberish? What guarantees that its generated by the official model?

This only works with maths - that is SHA-256 or other hash algorithms in a Proof of Work manner. The only thing that can't be spoofed is maths.

Reply on news.ycombinator.com