Run 100B+ language models at home, BitTorrent‑style

So, I believe 1 sec/token with Petals is the best you can get for the models of this size, unless you have enough GPUs to fit the entire model into the GPU memory (you'd need 3x A100 or 8x 3090 for the 8-bit quantized model).

I love this "bittorent" style swarms compared to the crypto-phase where everything was pay-to-play. People just sharing resources for the community is what the Internet needs more of.

Is there anything in the license that specifically forbids distributed usage? If not, you can run it on Petals and just specify that anyone using it must do so for research purposes (or whatever are the license terms)

Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.

Eschew flamebait. Avoid generic tangents. Omit internet tropes.

Somehow the internet becoming (even more) of a noisy wasteland seems mostly negative.

Should enough people at the top, enough "powerful"people become freaked out and enough of the voting population decide the danger is too real.

If America goes that way, basically all other countries will follow too. I don't buy this, "If we stop, China will keep going thing". I'm sure China has it's own concerns, and they're not 100% self-destructive.

1984, but real.

So I'd argue, you might actually be wrong. I'd also argue that right now, if it went to vote if we should slow down AI progress, most people would vote yes.

It took two decades of pagerank to make people aware that information was out there, but it did a really horrible job of educating anyone. Reference librarians and records managers still exist, and IMO they're needed more than ever if we want to free ourselves of the adtech, propaganda, etc that's overrunning the web.

We need the non-commercial web back.

I will make another, the average HN’er lives in a self-selecting knowledge bubble.

It matters a lot. Spam is easy to recognize and e.g. my current email client filters out dozens to hundreds of spam mails per day without any false positives. If you cannot distinguish spam from normal posts, this could even cause democracy to break. Unfortunately, there are strong anti-democratic forces in the world who want this to happen. In my humble opinion, this is the biggest threat to humanity right now because (unlike other threats) it's not hypothetical, it's going to happen.

(1) getting better is bad because you can enter the two words into Bing Chat or whatever to generate the same shit yourself, so you won't need them anyway, they only get in the way when you want to look for actual human-generated/curated content.

(2) getting better is obviously bad. Imagine most user-generated content turning into Quora-style ads or Amazon fake reviews, except with eloquence and bullshit knobs turned to 120%. Everything you read is coherent, convincing prose, you just don't know whether they're 100% false.

> you load a small part of the model, then team up with people serving the other parts to run inference or fine-tuning.

If multiple people participate in a fine tuning session, you have to trust all of them. You also have to trust everybody for inference too, but at least one of them can’t scramble the model.

Do each node earn points for supplying resources that can then be spend for greater query / process speed?

According to this excerpt, a node in the network doesn’t need load the entire model. Only a part.

But it's still decentralized, and decentralization drives competition in a way that traditional B2B contracts cannot. The fact that anyone on the planet who can afford a GPU or an ASIC can be a competitor is significant.

For example, an RX 6800 will generate ~$0.34 per day minus electricity costs if you mine with it. That's the true value of that card on a global decentralized market. But renting a similar cloud GPU will cost about $0.30 per hour. 95% of that cost could be eliminated with a decentralized market.

> Hivemind is a PyTorch library for decentralized deep learning across the Internet.

0: https://github.com/learning-at-home/hivemind

I don't think so. To simplify: You send out 1000 tasks, you perform them yourself, now you have 999 bad flags and 1 good one, you send out 10 tasks to the same including the good one, now you have 990 with 1 bad flag, 9 with 2 and 1 with 2 good ones, you continue sending tasks to the bad nodes and drop their response, if they send you a task you return garbage, you ask the good nodes (with say 100+ good flags) for their list of good nodes and test them one by one.

You could build a system where bad nodes have to return so many good responses before getting booted that the joke is on them.

Isn't the attack straightforward?

i) Take the model, freeze all the weights except the ones you expect to be responsible for

ii) Finetune to produce whatever output you are looking for.

iii) Profit. Or mainly just annoy people, but it could be funny.

The training of LLMs I believe has a well defined loss function with a measurable value on a specific input set. I actually already discussed this problem extensively with GPT-4 itself so I’ll leave you with parts of that discussion:

> Me: Do LLMs have a well defined loss function that they optimize over?

> GPT-4 Assistant: Yes, large language models (LLMs) have a well-defined loss function that they optimize over during the training process. The choice of loss function depends on the specific task the model is designed for and the architecture used. For LLMs like GPT-3, which are based on the transformer architecture and designed for natural language processing tasks, the most commonly used loss function is the cross-entropy loss.

> Cross-entropy loss measures the difference between the model's predicted probability distribution over the possible output tokens (words, subwords, or characters, depending on the tokenization used) and the true probability distribution, which assigns a probability of 1 to the correct output token and 0 to all other tokens. The goal of the training process is to minimize the cross-entropy loss, which essentially means making the model's predicted probabilities as close as possible to the true probabilities.

> During training, the LLM processes input sequences (e.g., sentences or paragraphs) and generates output token probabilities based on its current weights. The cross-entropy loss is computed using these probabilities and the true target tokens, and then the gradients of the loss with respect to the model's weights are calculated using backpropagation. The weights are updated using an optimization algorithm, such as stochastic gradient descent (SGD) or one of its variants like Adam or Adagrad, in order to minimize the loss.

> The process of iteratively updating the model's weights to minimize the loss function is what allows LLMs to learn patterns and structure in the training data, ultimately enabling them to generate coherent text, answer questions, or perform other language-related tasks.

> Me: Is it possible to restrict the training with a rules that says only these weights, say specific slices, are allowed to change, keep all other weights fixed? Is this how LLMs can be trained in a distributed fashion?

> GPT-4 Assistant: Yes, it is possible to restrict the training of an LLM to specific slices or subsets of weights while keeping other weights fixed. This is a common approach used in distributed and parallel training scenarios, where multiple devices or nodes work on different parts of the model simultaneously. This approach is known as model parallelism.

> In model parallelism, the model's weights are divided into smaller subsets or partitions, with each partition being assigned to a different device or node. During training, each device works only on updating the weights in its assigned partition while keeping the weights in the other partitions fixed. The gradients are computed for each partition separately, and the updates are performed locally on each device.

> Once all devices complete their local updates, they communicate their updated weights with one another to synchronize the model's state. This communication can be done through various methods, such as parameter server architecture or all-reduce algorithms. The key to effective model parallelism is balancing the computational workload among the devices while minimizing the communication overhead.

> Adapting this concept for the proposed proof of work and LLM training combination could involve assigning different subsets of weights to miners and having them work on updating these subsets. This would essentially turn the mining process into a form of distributed LLM training, with each miner contributing to a portion of the model's weights. However, coordinating and synchronizing the updates among miners in a decentralized environment, while maintaining the integrity and security of the blockchain, would be a significant challenge that needs to be addressed in the design.

But why does it have to be wasteful?

As I understand it you just need a hard enough computation to prevent the 51% problem.

Obviously if the goal was to compute something which had real value all the failed attempts would count as “provably wasteful” so would seem to fit the criteria.