Hacker News Reader

Comment by leshow | Hacker News Reader

What do you think Apple or Google does with your files after they are uploaded? Do you think they aren't encrypted and spread out over several machines?

View on HN

But with Apple and Google, I know my data won’t go missing because the p2p network decided to stop seeding my file. Torrents die all the time due to lack of seeding.

I believe that's what projects like Filecoin and Swarm are trying to solve. How do you incentivise long term storage of these files.

Torrent hosts do not have a monetary incentive to host files.

Filecoin (and other decentralized storage systems) provide monetary rewards and penalties for storing or failing to store files. This is introduced in Section 4.1.1 of this paper.

Exactly. As long as there's demand to keep hosting files in the network, there WILL be providers hosting your files. Even if a hosting company who hosts 5% of the network (that's a lot) decides to exit the and stop hosting, your files are spread between enough computers and you only need about half of them to be online to recover your files.

Unless some parts of your data happened to be replicated across multiple shards on that same hosting company's hardware. Then you would have a partial loss of data.

The bad thing is that you would not be able to hold them responsible.

> Torrent hosts do not have a monetary incentive to host files.

And on private torrent sites, where there is an incentive (usually non-monetary) to seed torrents, torrents really don't die all that often.

I think it's a problem of reputation, actually. Google and Apple keep stuff online because they promised they would, and there would be backlash if they turned it off all of a sudden. There's nothing stopping a random, anonymous "filer" (like miner) from stopping the upload stream.

To go even further, if you orchestrate several miners carefully (maybe computationally) they can together perhaps stop people from accessing certain files.

>my data won’t go missing because the p2p network decided to stop seeding my file

It'd blame Apple & Google for not being p2p seeds.

All those machines are owned by Google or Apple though. They're not completely random machines.

What's the problem with them being on random machines? I believe the purpose of Filecoin and other projects like Swarm is to incentives people to store the files long term and in some cases verify that those files are still being held.

What I don't understand is how this will guarantee the existence of these files. How do I know my files are secure? If my files are spread over several machines, and one or several go offline, are reinstalled, whatever. How do I know my files are secure?

They are most likely encrypted with your public key, no?

Presumably your data is duplicated to the point that all nodes containing any particular piece of data are very unlikely to be offline at a given time.

I haven't had a chance to dig into this paper yet, but for a similar system Swarm [0], there is a system of insurance and escrow on file chunks, and a system of verifying the hash of piece of the file to see if they still have its contents.

[0] http://swarm-gateways.net/bzz:/theswarm.eth/

The file is encrypted locally before uploading to the network. Then the file is sharded, and the shards are replicated among the nodes. Once a shard falls below a certain replication threshold, it will get copied to a new node.

There are other schemes that don't require a copy of each shard, allowing you to reconstruct the file if any subset of the nodes is online.

> allowing you to reconstruct the file if any subset of the nodes is online.

Any subset? Surely there's some minimum number of nodes (>1) that need to be online. Unless each node has a complete copy.

I don't know how Filecoin strengthens reliability, but Sia splits your file to 30 small pieces and distributes them randomly among hosts. You only need 10 of the pieces to recover a file. So, in effect, you rely on 10 out of 30 random hosts to be online at the time of recovery. The chance of more than 20 random hosts that are holding your file are offline at the same time is extremely small as long as there's enough diversity among hosters (particularly many different hosting companies and many individual hosters in different regions, so hosting isn't mostly "centralized" to one region or company).

I think Sia's method would prove to be extremely reliable, but again, it depends on the hosters. If the largest single hoster owns less than a third of the capacity of the network, even if they suddenly go offline, you should be able to recover your file since you rely only on a third of the 30 hosts who hold your file to be online.

Also, I think that the number of pieces you need to recover a file can be tuned, so if 33% would prove to not be reliable enough, Sia/Filecoin can tune it down to increase reliability.

Reply on news.ycombinator.com