Run LLMs at home, BitTorrent‑style
petals.dev
petals.dev
It's early but I've been working on hosting model weights in a Docker registry for https://github.com/jmorganca/ollama. Mainly for the content addressability (Ollama will verify the correct weights are downloaded every time) and ultimately weights can be fetched by their content instead of by their name or url (which may change!). Perhaps a good next step might be to split the models by layers and store each layer independently for use cases like this (or even just for downloading + running larger models over several "local" machines).
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
Surely in this case it would've been possible to comment about OP's work while leaving out the free backlink to your project. Just my 0.02
This is the part that raised my eyebrows.
Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"
I'm not sure what exact LORA/quantization settings would be ideal, but check out https://github.com/OpenAccess-AI-Collective/axolotl#config
For that 70B parameter model: an A100?
Source: have a $200 board in my computer right now with two full-length PCIe slots.
Meh. If you want 16GB of VRAM for several hundred dollars, can't you just pull a brand new 30-series off the shelf and have ten times more computing power than those old pascal cards? You'll even have more VRAM if you go for the 3080 or 3090. Admittedly, the 3090 is closer to $700 or so, but it should still make a P100 very sad in comparison.
Another curious contender is the decommissioned Nvidia CMP series GPUs from miners. For example, the Nvidia CMP 170HX basically uses the same Nvidia A100 PCB with its features downsized or disabled (8 GB VRAM, halved shaders, etc). But interestingly, it seems to preserve the full 1500 GB/s memory bandwidth, making it potentially an interesting card for running memory-bound simulations.
[1] Prices are so low exactly because most people don't. SXM-to-PCIe adapters also exist which cost $100-$200 - nearly as much as you have saved. It should be trivial to reverse-engineer the pinout to make a free and open source version.
...But it doesn't seem to be cheap. Not really worth it over a 4090 for the same price.
The main bottleneck to doing distributed training like this is the communication between nodes.
Consider the trivial example of training a network to distinguish between sample A and sample B. Give it a hundred As in a row and it just learns "everything is A". Give it a hundred Bs in a row and it relearns "no, everything is B". To train it to distinguish, you must alternate As and Bs (and not too regularly, either!)
Some inference frameworks came up with schemes for just this, and it was horrifically slow.
I have only a rudimentary understanding of neural nets but it doesn't seem crazy that the weights could be manipulated in such a way while preserving the utility of the model.
I ask because I think it would be useful to know which statements two LLMs of equal power agree on and which they disagree on. You could then map that backwards to differences in their training data (only feasible if the differences are small).
If instead two LLMs of equal power represent a missed opportunity to have one of greater power, and the disagreement analysis is prohibitively expensive to do, then that's a bit of a different world.
They're not composable in the sense that you can take these adaptation layers and arbitrarily combine them, but training different models while sharing a common base of weights is a solved problem.
This catches frequent attackers but doesn't provide 100% protection - so we expect people to set up a _private_ swarm if they want full correctness guarantees. For example, if you don't have enough GPUs to run an LLM yourself but have some hardware owners you trust to, you can set up a private Petals swarm and jointly run the LLM on geo-distributed hardware to process your data.
Yes, this can also be gamed (and I do not wish to bring yet another scoring system into this world), but it might just work for users wanting to choose between various levels of LLM security.
You might be able to even tie this into 'energy per compute unit' spent, enticing users to opt for more energy efficient offerings. Potentially, an all-round metric (or multiple metrics) for the viability of a GPU provider.
Will Petals incentives be based on crypto, blockchain, etc.?
No, we are working on a centralized incentive system similar to the AI Horde kudos, even though Petals is a fully decentralized system in all other aspects. We do not plan to provide a service to exchange these points for money, so you should see these incentives as "game" points designed to be spent inside our system.
Petals is an ML-focused project designed for ML researchers and engineers, it does not have anything to do with finance. We decided to make the incentive system centralized because it is much easier to develop and maintain, so we can focus on developing features useful for ML researchers.
What they are referencing, which is super cool and (IMO) criminally underused:
In fact, I can host a 13B-70B finetune in the afternoon if anyone on HN wants to test a particular one out:
is there a more canonical blogpost or link to learn more about the technical decisions here?
It is (IMO) a necessary and good change.
I just specified gguf because my 3090 cannot host a 70B model without offloading outside of exLlama's very new ~2 bit quantization. And pre quantized gguf is a much smaller download than raw fp16 for conversion.
It's what their proponent claim that they are useful for, yet there's no single instance of a successful blockchain project actually achieving this kind of resource-sharing goal.
> You have people all over the world with GPUs, some of them want to pay the others for use of them
The gigantic success of bitTorrent shows that humans as a group don't need to have monetary incentives to share their spare hardware. In fact, it's likely that trying to add money into the mix will just break the system instead of improving it: https://en.wikipedia.org/wiki/Overjustification_effect
Sia and Filecoin already work in this way to for people to share storage.
> In fact, it's likely that trying to add money into the mix will just break the system instead of improving it
This depends on the amount of money people are willing to pay for processing power. Volunteer contributions would be reduced, but the paid contributions could make up for it if the people who want to train their model pay enough to attract more people into the system and if those people can compete with conventional commercial offerings.
You'll notice that I said “successful” in my original sentence.
> This depends on the amount of money people are willing to pay for processing power. Volunteer contributions would be reduced, but the paid contributions could make up for it if the people who want to train their model pay enough to attract more people into the system and if those people can compete with conventional commercial offerings.
That's a very big “if”: the distributed nature of things is always going to make it more expensive than a traditional solution, especially if you need byzantine fault tolerance (which you need as soon as their monetary value to earn by cheating), the same way that a blockchain is orders of magnitude more expensive than a cloud KV store database, and by pushing the volunteers away you'll end up with a small pool of for-profit actors and these actors themselves likely would be better off if they provided their own cloud offering.
For instance filecoin only has a low thousands nodes, the average filecoin node has something like 10PB of available storage, the top three having 90PB each and making barely $1600 a day, which is $6.4 a year per TB.
> especially if you need byzantine fault tolerance
For storage this can be done much more efficiently with erasure coding and hashing.
For compute, reputation. A node with no reputation has all of its output verified (and so gets paid less). A node with a good reputation history only gets random spot checks, but fail a spot check and you're back to getting paid less, maybe even retroactively.
> For instance filecoin only has a low thousands nodes, the average filecoin node has something like 10PB of available storage, the top three having 90PB each and making barely $1600 a day, which is $6.4 a year per TB.
So it costs too much and it's too cheap?
The nature of something like this is low barrier to entry, so the high competitiveness is going to result in low prices. That's kind of the idea.
The result is going to be two main categories of supplier. One, huge nodes with economies of scale. These might take lower prices than some retail cloud offering, but they also don't have customer acquisition or support costs. Two, nodes with "free" storage, e.g. you built a media center which is already on 24/7 but still has a few TB of free space, so until you get around to using it yourself you'll take the however much in free money. In both cases because they have lower costs than competing providers.
It sounds like the network is providing several exabytes of storage for an extremely competitive price. How is that not a success?
More efficient that what exactly? It's still far less efficient than not having to hash and run erasure coding…
> For compute, reputation. A node with no reputation has all of its output verified (and so gets paid less). A node with a good reputation history only gets random spot checks, but fail a spot check and you're back to getting paid less, maybe even retroactively.
That only works if the attacker cannot make big gains from a single cheat after a period of building reputation. There's a reason why this isn't being used in the wild by blockchains…
> So it costs too much and it's too cheap?
Yes, it costs too much to operate, and it's too cheap as a product so operators are losing money. The only reason why there's an offering at all is that some people invested lots of money on hardware in 2021 when the token price was 50 times higher (but then the storage cost was prohibitive).
> It sounds like the network is providing several exabytes of storage for an extremely competitive price. How is that not a success?
Barely anyone using it despite a price so low that it doesn't even allow operators to break even, how is that supposed to be a success?
Large storage systems already do this. RAID is a type of erasure coding. Various enterprise storage systems use hashing for data integrity.
Distributing the data over a larger number of systems can actually be more efficient because the percentage of erasure blocks you need goes down as the number of independent devices increases. For example, if you only have two devices and want redundancy, you need a mirror and lose 50% of your capacity. If you spread the data across 200 devices you might achieve an even higher level of resilience by sacrificing only 25% of capacity so you can lose any 50 devices without data loss. You may get this down even further by periodically checking if a device is still available and replacing it, so the number of devices you can lose can be as low as the maximum number of devices you expect to lose simultaneously.
> That only works if the attacker cannot make big gains from a single cheat after a period of building reputation. There's a reason why this isn't being used in the wild by blockchains...
Blockchains are a different thing.
If you have a GPU and to start out with everything you produce is checked, you get 50% of what the customer pays. If you have a good reputation then only e.g. 1 in 10 of your output is checked, so you get ~90% of what the customer pays, but to do that you have to accept 50% for e.g. 100 transactions.
You can now defect, but you have a 10% chance of being detected each time, so you can expect to only get to do it 10 times. So you get a 90% payment ten times without doing work, then have to go back to getting a 50% payment 100 times. 10 times 90% is way less than 100 times 40%, so if you do this you lose money.
> Yes, it costs too much to operate, and it's too cheap as a product so operators are losing money.
Do you know what the overhead of the network actually is? Trying to put it together from multiple sources seems to imply that miners get paid ~$8/TB/year but storage costs ~$2/TB/year. Which I assume I'm doing wrong somehow, because it would imply negative overhead and therefore a huge arbitrage opportunity.
I'm guessing the real number is less than 50% overhead, because there are obvious ways to do it at least that efficiently, but even that isn't huge when you can avoid expenses for marketing and customer support. Which implies that the problem is this:
> The only reason why there's an offering at all is that some people invested lots of money on hardware in 2021 when the token price was 50 times higher (but then the storage cost was prohibitive).
Which is a self-solving problem. The unprofitable providers go out of business until the price makes it profitable. But that seems like it should happen quicker than this if the profitability isn't there, because storage is fungible. Even if you bought a bunch of drives to do this when the price was higher, you could sell them and go put the money in a traditional investment. Or if you're speculating on the value of Filecoin going up, sell your storage and use the money to buy Filecoin. So the people still doing it are presumably turning a profit even at current prices, whether through economies of scale or because they had "free" storage to use.
> Barely anyone using it despite a price so low that it doesn't even allow operators to break even, how is that supposed to be a success?
It causes very inexpensive storage to be available, which is useful.
> Distributing the data over a larger number of systems can actually be more efficient because the percentage of erasure blocks you need goes down as the number of independent devices increases. For example, if you only have two devices and want redundancy, you need a mirror and lose 50% of your capacity. If you spread the data across 200 devices you might achieve an even higher level of resilience by sacrificing only 25% of capacity so you can lose any 50 devices without data loss. You may get this down even further by periodically checking if a device is still available and replacing it, so the number of devices you can lose can be as low as the maximum number of devices you expect to lose simultaneously.
You're mixing things up so bad it's hard to correct… For starter regarding filecoin the number of nodes you must expect to lose is “almost all of them” because they can stop operating if the economics isn't even good enough to cover their OpEx (They seem to be fine not covering the CapEx for now, but who knows for how long). It's almost like putting all your data in a datacenter owned by a nearly broke provider: if they go bankrupt you're screwed so you need a plan B.
> You can now defect, but you have a 10% chance of being detected each time, so you can expect to only get to do it 10 times. So you get a 90% payment ten times without doing work, then have to go back to getting a 50% payment 100 times. 10 times 90% is way less than 100 times 40%, so if you do this you lose money.
Again, you're mixing things up. The problem isn't that a node could defect and not do the work (this is assuming non-byzantine fault tolerance), the problem is that a node could voluntarily fuck up the calculations when/if it advantages them. And it could be far less than 10% of the time while still being a nuisance. I don't need to fuck up 10% of the back-propagation calculations in a neural-network training to make it completely unusable/to make the person training it spend way more resource than they should in the training process (which gets me more usage as a node operator).
Adversarial threat modeling is hard, I work with people who do this on a daily basis and I can clearly tell you're oversimplifying things a lot.
> Which is a self-solving problem. The unprofitable providers go out of business until the price makes it profitable. But that seems like it should happen quicker than this if the profitability isn't there, because storage is fungible. Even if you bought a bunch of drives to do this when the price was higher, you could sell them and go put the money in a traditional investment. Or if you're speculating on the value of Filecoin going up, sell your storage and use the money to buy Filecoin. So the people still doing it are presumably turning a profit even at current prices, whether through economies of scale or because they had "free" storage to use.
And yet, it hasn't self-solved itself since 2021… The problem is that node operators get paid roughly what they spend in OpEx, and their capital is essentially illiquid[1] so there's no good reason to stop operating. Of course now the only reason for them to keep invest is the hope that the token price increases again, but because this is crypto it is fueled by the “fantasy of the bull run”, not by an expected uptick in usage (which interestingly enough isn't happening even though the storage price is very cheap).
> It causes very inexpensive storage to be available, which is useful.
Yet barely anyone uses it, which empirically question its “usefulness”.
[1] and I don't get were you got the idea that storage was “fungible”, the failure rate going up exponentially over time makes storage a poor fit for the second-hand market, especially if people know that you've been running stressful proof of space-time on it, and if you're trying to fire-sale a Petabyte of storage, chances are high that people will figure that out
You don't expect to lose "almost all of them" at the same time. Even if the price crashes, you would expect capacity to go down over a period of days or weeks, not minutes. And then if a fraction of the data is lost but is less than the number of erasure blocks, you promptly reconstruct it and put it on a different node.
Meanwhile you would expect an equilibrium here. The price going down forces some providers out of the market, but providers leaving the market brings the price back up. As long as the customer is offering as much as some providers need to stay in the market, somebody is hosting the data. That only stops if the customer won't bid what the providers need to get, at which point the customer transfers their data out of the system so they can stop paying more than they're willing to.
Now, you can screw this up if you make your system sufficiently convoluted so the price signal doesn't make it from the customer to the provider or vice versa, and I'm not familiar enough with the specific implementation in Filecoin to comment, but screwing that up isn't inherently necessary for this category of system.
> the problem is that a node could voluntarily fuck up the calculations when/if it advantages them.
Which is why you duplicate some of them at random, and don't tell them when you're going to do it. The calculations are deterministic. If you distribute one to two random nodes and they don't get the same result, but they've each signed their own result, now you know one of them defected and can prove which one by doing that calculation yourself or doing some other potentially expensive operation that only happens when there is an inconsistency. At which point the defector is found out, you can prove it, and their reputation is in ashes.
> Yet barely anyone uses it, which empirically question its “usefulness”.
It seems to be storing more than a exabyte of data for someone.
> their capital is essentially illiquid[1]
> [1] and I don't get were you got the idea that storage was “fungible”, the failure rate going up exponentially over time makes storage a poor fit for the second-hand market, especially if people know that you've been running stressful proof of space-time on it, and if you're trying to fire-sale a Petabyte of storage, chances are high that people will figure that out
It's fungible because a used hard drive is a commodity product with a wide customer base. That new ones cost more than the used ones doesn't make it not a commodity; you could have bought the used ones to begin if you're content to continue running them at their current age.
Let me know if you're aware of some place you can buy working >=16TB drives, used or otherwise, for less than ~$100 each in 2023.
Bringing the XIXth century equilibrium economics here is kind of hilarious, when it has been far out of the equilibrium price for the past two years. Node runners are already losing money, and have been doing so since the end of the bull run. The value of the FIL token isn't so much derived from an equilibrium in the supply and demand for storage, it's driven by the supply and demands of the coin on the crypto market, and if nodes start to give up in any meaningful fashion over the course of a few days or weeks, the crypto market will likely negatively react, driving the price of the token even lower. In the crypto markets, Keynes' animal spirit is in charge, nobody makes rational utility calculation.
> At which point the defector is found out, you can prove it, and their reputation is in ashes.
If I can make more money out of a single adversarial attack than it costs me to build up reputation, then who cares, I'll be doing it over and over again any day.
> It seems to be storing more than a exabyte of data for someone.
It is in fact storing a exabyte of “data” for “someone”. Compare that with BitTorrent, that was used by everyone and their mom before governments started to fight it. BitTorrent was voluntary only and was a massive success. Filecoin is for profit and a failure.
> It's fungible because a used hard drive is a commodity product with a wide customer base.
Try and sell 10PB of PoST-worn-out hard drives and see how long it takes. It's far from liquid.
The reason for this is that the price of FIL was initially too high for the amount of customer demand for storage it currently has, resulting in oversupply. But you only lose data as a result of sudden undersupply. If the network could lose 90% of its capacity over a month and still store all of the data it currently does, and then that happens, so what?
Whereas if it actually lost enough capacity to create scarcity given the existing demand for storage, then demand for storage would drive the price of the coin back up, right?
> If I can make more money out of a single adversarial attack than it costs me to build up reputation, then who cares, I'll be doing it over and over again any day.
How are you going to do that with AI training or something? As soon as you get caught once, people go back and retroactively verify everything you've previously done, and then you not only lose any payment received for each calculation you forged, the model you screwed up gets recomputed using the money you didn't get to keep or had to stake in order to be trusted to do computations with lower frequency verification.
> Compare that with BitTorrent, that was used by everyone and their mom before governments started to fight it. BitTorrent was voluntary only and was a massive success.
BitTorrent is a great success for large, popular data. It's pretty much useless for storing anything with a low number of downloads.
> Try and sell 10PB of PoST-worn-out hard drives and see how long it takes. It's far from liquid.
Put functional 16TB hard drives on Amazon and eBay for $99.99. See how long they last. I'd guess less than six months before you've sold 10PB worth.
If the network lose 90% of capacity over a month, you'll hear about grim the future for FIL is, on every crypto newsletter. And the price would tank even more. And if the network already lost 90% of its capacity it means that the economics is already very bad for node operators, so any worsening is likely to get even more node leaving the ship. Crypto going do dust because of crowd dynamics isn't completely unheard of…
> As soon as you get caught once, people go back and retroactively verify everything you've previously done,
How can they link me to my previous identity though… I'd just discard the previous wallet after having drained the available funds and restart from a clean state.
> the model you screwed up gets recomputed using the money you didn't get to keep or had to stake in order to be trusted to do computations with lower frequency verification.
The stacking must end at some point, and given that I can do damage with only a fraction percent adversarial computation, I can just make sure that my probability to get caught during the stacking time isn't enough for it to get a negative expected value.
Your scheme is pathologically broken, and that's no surprise, you're not going to invent a billion dollar winning multiparty computation model as an argument on HN…
> Put functional 16TB hard drives on Amazon and eBay for $99.99. See how long they last. I'd guess less than six months before you've sold 10PB worth.
You'll need to send roughly a thousand of them, without getting bad reputation from all the disks that will break soon after the buyer receives it (because on that amount, and given the state of the disks, a lots will). Also, you're not really disagreeing with my assessment, 6 month is pretty illiquid by investment standard: it's even less liquid than real estate!
But the supply of storage goes down, which the storage buyers now need to outbid each other for, so they need to buy the coin. I'm assuming it's also possible for the price of storage in FIL to go down as the price moves. If $1 US is 100 FIL but now providing 1TB/year of storage yields hundreds of FIL, you still earn several dollars US per TB stored.
> How can they link me to my previous identity though… I'd just discard the previous wallet after having drained the available funds and restart from a clean state.
No reputation is the same as bad reputation. To have a good reputation you have to engage in a large number of transactions which are less profitable to you because they're undergoing 100% verification. Building a good reputation allows you to make higher margins, which is valuable and therefore costly to sacrifice.
You can't transfer funds you've staked against your reputation until the buyer has had a reasonable amount of time to try to prove you defected.
> The stacking must end at some point, and given that I can do damage with only a fraction percent adversarial computation, I can just make sure that my probability to get caught during the stacking time isn't enough for it to get a negative expected value.
Suppose you have a good reputation so you only undergo verification 10% of the time at random instead of 100% of the time. You also have to hold 20 times your revenue from this transaction as collateral during the verification window, or however much is necessary to more than compensate the buyer and punish you in the event that you defected.
Now if you defect you have a 10% chance of losing 2000% of your payment. This has a negative expected value. Meanwhile it's now public that you defected and every other buyer still in the verification window is going to go back and verify 100% of their transactions with you, causing you to have a 100% chance of losing 2000% of your payment for those transactions if you defected.
> You'll need to send roughly a thousand of them, without getting bad reputation from all the disks that will break soon after the buyer receives it (because on that amount, and given the state of the disks, a lots will).
The annual failure rate for ~6 year old hard drives is ~2%. Presumably the failure rate over six months is about half that, and you have plenty of other functional drives to send replacements to satisfy the ~1% of customers who got unlucky.
> Also, you're not really disagreeing with my assessment, 6 month is pretty illiquid by investment standard: it's even less liquid than real estate!
That's only because you're trying to sell 10PB of hard drives. It's like saying shares of stock are illiquid because if you want to sell ten billion dollars of shares in the same company it might not be advisable to do it all on the same day.
And even that you could still do, if you want to solicit a large buyer, which in this context would presumably be some kind of data center.
But even supposing that it would take six months, what's your reasoning for why it has already persisted for longer than that period of time then?
This is just a balance between how much you win, and how much it costs you. If I double my earnings for sub 1% chance of being caught, then you need to have a very expensive reputation build-up to compensate for that, and this is going to put a big burden on legit providers who want to enter the system, making it even easier to cheat.
> You also have to hold 20 times your revenue from this transaction as collateral during the verification window, or however much is necessary to more than compensate the buyer and punish you in the event that you defected.
Same as above: the higher the stacking is to fend of cheaters, the less attractive it is to legit players. Also, with your scheme the “verification window” doesn't matter, since you're not going to catch me after the fact: you're going to catch me iif my adversarial transaction is being checked.
> Now if you defect you have a 10% chance of losing 2000% of your payment.
Not if me cheating allows me to make even just 223% of the legit payment.
As I said before, you're not going to design a billion-dollar scheme in this HN discussion…
Edit: I just realized that your scheme would be even more penalizing to legit node than I thought: with one random bit flip, the node would lose all their stacking and all their reputation. Talk about an expensive cosmic ray! (Or an attacker could even voluntarily send rowhammer workload to legit node in order to destroy their reputation and stacking, reducing supply and hence increasing their own margin). And I'll say it again: this stuff is HARD and you're very unlikely to find a working solution on your own in this discussion!
> The annual failure rate for ~6 year old hard drives is ~2%.
Not if you've spend the said 2 years stressing the drive in a PoST scheme. There's a reason why these schemes break the manufacturer's warranty …
> That's only because you're trying to sell 10PB of hard drives. It's like saying shares of stock are illiquid because if you want to sell ten billion dollars of shares in the same company it might not be advisable to do it all on the same day.
You'll be able to sell them. You'll take a haircut (likely less than the 70% you're talking about when reselling old hard drives), but you'll sell them in the same day anyway.
> And even that you could still do, if you want to solicit a large buyer, which in this context would presumably be some kind of data center.
Good luck selling worn-out hardware to a data center!
> But even supposing that it would take six months, what's your reasoning for why it has already persisted for longer than that period of time then?
Hodl to the moon (AKA sunk cost fallacy)… They have an illiquid asset, have swallowed the cost of the capital investment (i.e. they had little to no leverage on it) and no need to fire sell it. Also the 6 months is just your optimistic hypothesis…
But there is also no need to make the chance of being caught so low, because a single digit percentage of overhead is completely reasonable while still providing a significant chance of being caught.
And you could scale the verification rate with reputation, so a 1% verification rate is possible but takes a very long time, whereas a 10% verification rate is more than ten times easier to get despite still not being a prohibitively high amount of overhead.
> Same as above: the higher the stacking is to fend of cheaters, the less attractive it is to legit players.
For things like GPU computation, you're going to do a unit of computation over a matter of minutes with verification taking the same amount of time or being done in parallel, and so you do many units of computation a day.
It's not that unreasonable to ask someone to put a month's earnings at stake at any given time, which is about what you get with a 24 hour verification window and a 7% verification rate.
> Also, with your scheme the “verification window” doesn't matter, since you're not going to catch me after the fact: you're going to catch me iif my adversarial transaction is being checked.
You're not the one who chooses whether to verify it. If other people are verifying 10% of your work but then someone catches you cheating, they can prove to the others that you cheated and then everyone goes back and verifies 100% of your work which is still in the verification window -- at your expense -- and you lose even more if you cheated more than once.
> Not if me cheating allows me to make even just 223% of the legit payment.
The thing this is preventing is you claiming to do some work but actually not, e.g. someone wants to use your GPU for AI but you don't even have a GPU and just return random numbers. To know if your result is right they would have to do the same computation again so they can compare them, which doubles the cost, or more if they want higher assurances against collusion. And then they're not willing to pay you as much because some of their money has to go to that.
Also, it's 223% plus the cost of provably damaging your reputation.
> I just realized that your scheme would be even more penalizing to legit node than I thought: with one random bit flip, the node would lose all their stacking and all their reputation. Talk about an expensive cosmic ray!
You can set the penalty to whatever is necessary to deter cheating at that level of verification. It doesn't have to be that high, but it can be that high if you need it to be without imposing an unrealistic amount of overhead.
And providers who don't want to be penalized for doing the calculation wrong should operate reliable hardware with functioning error correction. This is not a bad thing to incentivize.
You might also weight the reputational harm. If you get caught once, your reputation will be harmed and lots of people will be rechecking your recent results to see if you tried to screw them too, but if it's an isolated incident you only take a small hit. Whereas if you get caught repeatedly, well, you might as well just start over.
> Or an attacker could even voluntarily send rowhammer workload to legit node in order to destroy their reputation and stacking, reducing supply and hence increasing their own margin
At which point the node is at least as likely to crash as sign an invalid result, which is already a denial of service attack you have to mitigate. For example by using ECC memory and terminating workloads that induce detectable ECC errors instead of continuing them until they induce an undetectable one and crash the machine or cause it to sign a corrupted result.
(Also, rowhammer is a huge problem and almost nobody is actually mitigating it effectively for anything. Someone needs to come up with a generic solution for it before someone else starts using it for widespread exploitation or we're going to have a bad time regardless of what kind of reputation systems are in use.)
> Not if you've spend the said 2 years stressing the drive in a PoST scheme. There's a reason why these schemes break the manufacturer's warranty ...
Do you have some data to back up this claim? Drives are routinely used for heavy database workloads and reliable drive models still last for multiple years.
It seems evident that they're at least reliable enough to continue operating under that workload since that's what they have done instead of failing en masse and causing the storage capacity of the network to decline, given the assumption that the price is too low to justify anyone replacing them.
> You'll be able to sell them. You'll take a haircut (likely less than the 70% you're talking about when reselling old hard drives), but you'll sell them in the same day anyway.
You can sell the hard drives the same day too if you're willing to provide a sufficient discount from the market price. But there is rarely a good reason to do this, because the discount you'd have to provide is more than the time value of money in spreading the sales over somewhat more time.
> Good luck selling worn-out hardware to a data center!
Data centers run hardware until the resource consumption in power and space exceeds the cost of newer hardware, or until it dies. Reliability is just a number in an equation that tells you how much redundancy you have to operate with.
> Hodl to the moon (AKA sunk cost fallacy)
Those are two different things. If they want to hold the coin they'd have more of it to hold if they sell their hardware and use the money to buy the coin.
More likely they're expecting that others will quit and lower the supply to fall in line with demand (or more optimistically that demand will increase) so they can go back to making a profit, but since they all have that incentive they hodl until somebody blinks first, and lose if nobody does.
In the meantime their possibly irrational optimism provides for cheap storage.
Edit: To clarify, I'm not suggesting linking these Petal "tokens" to any payment system. I'm talking about, in general, calls to clusters of machine learning models, decentralized or not, will likely use crypto payments because it gives you auth and a means of payment.
I do think Petal is a good implementation of using decentralized compute for model use and will likely be valuable long term.
>People who run inference and fine-tuning themselves get a certain speedup if they host a part of the model locally. Some may be also motivated to "give back" to the community helping them to run the model (similarly to how BitTorrent users help others by sharing data they have already downloaded).
>Since it may be not enough for everyone, we are also working on introducing explicit incentives ("bloom points") for people donating their GPU time to the public swarm. Once this system is ready, we will display the top contributors on our website. People who earned these points will be able to spend them on inference/fine-tuning with higher priority or increased security guarantees, or (maybe) exchange them for other rewards.
It does seem like they want a sort of centralized token however.
ImportError: cannot import name 'get_full_repo_name' from 'huggingface_hub' (~/.local/lib/python3.8/site-packages/huggingface_hub/__init__.py)[0] https://github.com/bigscience-workshop/petals/wiki/Launch-yo...
The Petals part was abstracted away from me. I had a normal experience writing code.
I don't have the project listed anywhere. Don't really know what happened to it. But, it was mainly some five or so guys spearheading the thing.
This is a good idea, as splitting up llms is actually pretty efficient with pipelined requests.
So, pretty much every other consumer PC available? Those losers.
Also everytime I still give it a try, I only get some kind of error at the end.
Edit: Here we go. Literally the first line that it wanted to execute: "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. tensorflow-metadata 1.14.0 requires protobuf<4.21,>=3.20.3, but you have protobuf 4.24.3 which is incompatible."
p.s. from experience instruct-finetuning falcon180b, it's not worth using over llama2-70b as it's significantly undertrained.
We developed Petals for people who have less GPU memory than needed. Also, there's still a chance of larger open models being released in the future.
And the inference is pretty inefficient. Pooling the hardware would achieve much better GPU utilization and (theoretically) faster responses for the host's requests
Here I'm assuming that Petal uses a large number of small, heterogenous nodes like consumer gpus. It might as well be something much simpler.
For inference? Yeah, but its still better than nothing if your hardware can't run the full model, or run it extremely slowly.
I think frameworks like MLC-LLM and llama.cpp kinda throw a wrench in this though, as you can get very acceptable throughput on an IGP or split across a CPU/dGPU, without that huge networking penalty. And pooling complete hosts (like AI Horde) is much cheaper.
I'm not sure what the training requirements are, but ultimately throughput is all that matters for training, especially if you can "buy" training time with otherwise idle GPU time.
in fact, if they did this so that it followed the sun so that the vast majority of it was powered by daylight Solar PV energy I wouldn't even be upset by that.
you Americans don't know how good you have it...
Edit: Oh hey, they did it.
Human: what is the time?
The time is 12:30 PM.
Human: are you sure?
Yes, I am sure. The time is 12:30 PM.^</s>^<s> I'm a young {...}
Apparently, the Colab code snippet is just too simplified and does not handle </s> correctly. This is not the case with the full chatbot app at https://chat.petals.dev - you can try it out instead.