Marketing claims, meh. It gives normal people the wrong impression.
You can’t parallelize your query because it’s sequential. I think people will be willing to wait the ~200 sec necessary to get 200 words, but it’s best to be up front about this limitation.
Also abuse is a problem. Once 4chan realizes they can poison the distributed model, they’ll have a field day. But maybe it’s too much effort for too little reward that trolls won’t bother.
> Q: Does Petals guarantee that model outputs are correct?
> Not by default. A faulty or malicious server could give you incorrect outputs. There are two things you can do about this:
> - Verify outputs. Send some of your tensors to two or more peers and check that the answers match.
> - Set up a private swarm. You can launch your own swarm hosted by people and organization you trust, who are authorized to process your data.
> In future, we plan to implement an automatic verification and a reputation system, so that clients can select servers that they can trust.
In turn, "parallel inference" refers to the high-throughput scenario when you generate multiple sequences in parallel. This is useful when you process some large dataset with LLM (e.g. run inference with batch size of 200) or run a beam search with a large beam width. In this case, you can actually get the speed of hundreds of tokens per sec, see our benchmarks for parallel forward passes: https://github.com/bigscience-workshop/petals#benchmarks
If you have another wording in mind that is more up front, please let us know, we'd be happy to improve the project description. Petals is a non-commercial research project, and we don't want to oversell anything.
Do each node earn points for supplying resources that can then be spend for greater query / process speed?
> you load a small part of the model, then team up with people serving the other parts to run inference or fine-tuning.
If multiple people participate in a fine tuning session, you have to trust all of them. You also have to trust everybody for inference too, but at least one of them can’t scramble the model.
I do love the general direction, and I think it's inevitable that training will move to be more decentralized like this. It's also the best chance we have at disrupting the centralization of "Open"AI and their ilk. I say the earlier we figure this out, the better, but it's not an easy problem to solve cleanly. And, not to be that guy, but maybe we could add some cryptocurrency incentives to the mix... conveniently enough, the crypto miners already have the GPUs ready to go!
- https://github.com/bigscience-workshop/petals/wiki/Security,...
- https://github.com/bigscience-workshop/petals/wiki/Launch-yo...