What Sets Us Apart: Filecoin’s Proof System | Hacker News Reader

What Sets Us Apart: Filecoin’s Proof System

filecoin.io

53 pointsConfiks65 comments

Like so many cryptocurrency projects, Filecoin just seems like a solution in search of a problem more than anything else. What exact problem is Filecoin trying to solve that's not possible with current infrastructure? It seems like the decentralization adds a lot of overhead and complexity for no benefit.

The main argument that I hear is that somehow Filecoin will be cheaper than centralized competitors. However, services such as Backblaze are already quite efficient. Why exactly would costs be cheaper and why can't Backblaze do those same things and overtake Filecoin?

They needed to compensate providers of storage for IPFS. The challenge is how to prove that a provider has your block and that they aren't double dipping, ie claiming redundancy when they only are storing one copy.

That's pretty neat. I wonder if similar could be achieved for processing power. Distribted hosting ftw.

that's what https://golem.network/ is trying to do.

Exactly. IPFS has millions of end users and hundreds of apps storing data on the decentralized network, creating demand for decentralized persistence of that data. Current solution is to either run your own IPFS node or pay a pinning service to run a node persisting your IPFS data for you - but even better to have a highly resilient marketplace of storage nodes that can store redundant copies!

Where are you getting “millions of end users” from? A quick google search doesn’t return anything, and that seems a lot for a relatively unknown project

IPFS is by no means 'relatively unknown'. It is used in dozens of projects. See: https://github.com/ipfs/awesome-ipfs

I think parent was saying the projects using IPFS are relatively unknown, so where are the millions come from?

I have the same question.

In contrast, GUN, has 20M+ downloads/month from known sources: Internet Archive, HackerNoon, etc. (https://github.com/amark/gun see jsdelivr download stats).

lol it's not a HN post about a decentralized project without marknadal shitting on it and shilling GUN. got a running bet it always happens, and you don't disappoint <3 -- keep it up!

Yeah, Its pretty much a staple. Despite repeated attempts by the mods to get it to stop.

IPFS, a YC company, is trying to censor Open Source competitors? Interesting. What are you scared of?

I'm not trying to censor anything, lol. Just referencing many hilarious moments in your comment history where the HN mods tell you to quit spamming, and here we are again with more off topic mark spam ;)

Links?

Gun is currently a pile of shit. Anyone saying otherwise hasn't used it. The sole founder and developer is a very nice person, but it's the reality of Gun tech right now.

Thanks for the comment.

Could you expand further what didn't work? We have some pretty heavy load in-production systems running. Definitely some rough edges still, so I'd like to hear what problems you had, so I can focus on fixing them in the future.

Thanks again.

It was a month ago I checked it out and it just wasn't usable - i do not remember the details but even basic things did not work. I had a json dataset i just wanted in a local db, and that was proving challenging for simple operations is what I remember, didn't even get into the p2p side yet. If I remember correctly, it wasn't returning accurate results when queried.

try creating a youtube video of getting it to work from scratch, reading and writing data and i think you will see the problems. Or maybe ask a friend that is less familiar than yourself to try it, the problems seemed rather obvious as it just didn't work for a basic use case.

Thanks for the reply.

If you ever have time, it'd be great to snag whatever code or data you had, to create a replicable test case for us to make a fix against. Sorry about the wasted time you experienced, I hate that.

I went through that list and can’t recognize any of those names. Most of them seem clones of “<website> but on ipfs”.

https://en.wikipedia.org/wiki/Everipedia alone is 3M monthly active users.

How many of those users connect via IPFS protocol though? Because I bet overwhelming majority of them use https website, and do not interact with IPFS in any way.

Presumably they think decentralization is a benefit in its own right.

Technical complexity isn't the only dimension.

Surveillance capitalism and privatized censorship are byproducts of data centralization.

Cryptocurrency has a political aspect. Why view it only through a lens of technical complexity?

If a decentralized system is using spare resources that are literally free it could theoretically be cheaper than a business that pays for hardware. It will be interesting to see if that works in practice.

One issue is that you are always paying for power. Some back of the napkin calculations indicate that Backblaze's prices are low enough that it's not worth it to compete unless you are getting more than 1 terrabyte per 30 watts as otherwise the power costs alone would kill you. That's going to be a pretty hard target to hit with "spare free resources" (not to mention hardware degradation and capital costs).

You're always paying for power regardless of whether you're offering your unused drive space on a distributed system.

Hard drives consume less power when idle, like CPUs.

I imagine that extra power consumption has to be less than the combination of non idle HDDs and all of the other infrastructure at backblaze, no? And they're still going to be idle most of the time.

And there's the added likelihood that people providing hdd space on filecoin are not going to even measure the impact of extra power consumption, they're just going to notice the revenue. (Not that this is a good thing, if someone was losing money, they should know)

Agreed.

Cloud storage is a commodity race to the bottom. Cloud providers want you to use their storage systems because it means that you’ll also use their more profitable associated services in order to save money on egress. Accumulate enough data, and you’re locked in because of the high cost to migrate. Because IO dominates so many workloads, you already get a bunch of bytes for cheap when you are running a cloud storage provider. It’s already dirt cheap and potentially a loss leader (I suspect).

So, I have a hard time seeing how “spare free resources” are going to be competitive when you factor in the network, capital costs, power, etc. Then there’s some unknown legal risk that you are e.g. accepting money to distribute child porn to people, just like there’s a legal risk associated with running a Tor node (not commenting on whether this is just or not).

Just my opinion. Not speaking on behalf of anyone here.

Storing and serving files a lot cheaper and more reliably than current cloud providers seems like something people actually want.

Yes, but does Filecoin actually get you there? And if so, how and why?

Right but you actually have to be able to do that, and introducing inefficiency in the form of blockchain doesn't seem like a way to make anything cheaper.

I don't want to start a business that pays a cloud service for storage. I want to write an application and set it free on the internet to run all by itself.

Like other cryptos, Filecoins founders don't get rich off existing and usually better solutions.

Founder of another decentralized storage product here:

Decentralized storage is going to replace the cloud. We have a working platform today with thousands (and growing) of users which is cheaper, faster, and more reliable than the centralized alternative.

One of the major challenges of the cloud today is deplatforming and loss of ownership, two things that are squarely handled by decentralization. It opens the door to innovation and allows builders a lot more freedom in how they approach solving modern problems.

Decentralization also ends up being a huge element in cost reduction. Our platform uses dozens of hosts at one to store and serve data, which means any given host only needs about 95% uptime. The network is completely open and anyone with good infra can join at any time, making revenue, and competing, which drives prices down. We're currently at 1/10th of Amazon.

Decentralization also makes it possible to build apps that don't shut down. Users today have to worry about the health of a parent company when choosing a product, with many people having been displayed by a "Google RSS Reader is shutting down" message. That problem disappears with decentralized infra.

It may take some time for the rest of the world to catch on, but decentralization is just a better way to build apps.

>> Our platform uses dozens of hosts

>> We're currently at 1/10th of Amazon.

Curious, where is the "1/10th of Amazon" coming from?

https://siastats.info/storage_pricing

What sets us apart: Inefficiency and a higher cost basis than any cloud storage providers due to our devout adherence to blockchain.

Whats missing is proof of delivery, and thats the most important thing. I'm not just paying AWS to store my files, I'm paying them to deliver my files.

Maybe this could work as an alternative to glacier. But even then, I doubt it. I'd take Amazon with an SLA any day of the week.

Correct, this is what we solved for.

I gave a presentation at Berkeley a few years back, right before Bram Cohen (BitTorrent) presented his Chia proof, Proofs of Space and Time (before IPFS ripped the name from Bram).

https://www.facebook.com/BlockchainatBerkeley/videos/2006069...

Sorry, who is "we"? And got a writeup anywhere?

https://web.stanford.edu/~nadal/A-Decentralized-Data-Synchro...

https://era.eco/

http://axe.eco/

That's the other way you get paid in Filecoin, payment channels for retrieving files. Although it's not technically "mining" it's easier to think about Filecoin having two types of miners: Storage and retrieval. Since it's incentivized there's a good chance you'll get your files back quickly if you're willing to pay more or if it's a common file.

As long as Filecoins is traded on a market, and not tied to actual cpu/bandwidth/storage costs, then if will ALWAYS be more expensive then AWS (and alternatives).

Not sure how thats true, If you and I can negotiate a price we are willing to pay for a given thing, it doesnt matter what the token is tied to or traded on.

If I go out and spend $20,000 on a a 480TB file server and I were charging AWS prices ($0.023/GB), I could get $11,040/ month. After two months my investment would have paid for itself. I'm not sure why you think markets are inefficient and somehow AWS is superior, but folks could be getting paid 10x less than AWS and still making money on their investments.

Inversely, people could be storing their files for 10x cheaper than on AWS.

Because Filecoin being a tradable asset as well as a utility confuses how it's valued. If it rises quickly, it's better to sell then spend. If it drops, it's bad for those running servers. Both ups and downs are being driven by forces which have nothing to do with cost of services, that's the problem. Unless they fix the price of Filecoin against the USD to make it a real utility, it's a get rich quick scheme.

You see Sia's Skynet announcement yet? It is the only technically 100% decentralized product and Downloading stuff from skynet is fast. check out their announcement that is on skynet.. 4k vid in there showing the speeds. It is impressive. https://siasky.net/GADwbULtGj2NuBXJrQPBkivr6etO5uD4BSToixUUS...

It seems that Filecoin fundamentally relies on a proof-of-work type system called "sealing." The article calls it "an intense amount of work" that you must do in order to host a file. Does anyone have any estimates or comparisons for the CPU/energy consumed?

Its currently looking at about 4 hours of single core CPU time, plus 2-3 hours more of using many cores for a 32GB sector.

The difference between sealing and an actual proof of work based network here is that once you've sealed a sector, you don't have to keep computing anything that expensive. If you seal 1TB and thats 1% of the network, you should win 1% of the blocks (ignoring network growth) without further sealing work required. I tend to think about it like a (calculus) derivative of PoW.

What's the purpose of sealing?

If I'm understanding it correctly, it's to ensure that you've actually set aside that disk space for the Filecoin network (instead of using it yourself).

Sealing ensures that miners have a unique replica of a given piece of data on their disk. Since we're basing block producer power on storage, we need to protect against a number of attacks such as people generating the data on the fly, or deduplicating data on disk (claiming to have 100TB but only having a single 1TB file 100 times).

Cool!

So then, dumb question. Can I delete the file, keep the "seal", and keep winning 1% of the blocks?

If yes, then that seems bad for the system.

If not (hopefully), I'd like to also understand what's the ongoing compute cost of using the seal plus file together.

I understood from the article that sealing is something you do if you are publishing a mirror of a file. Is that right?

Feel free to get bored of this conversation and link me to a paper or something. :) I'm grateful for the reply you gave above already.

Nope, thats where the second Proof comes in. PoSt (Proof of SpaceTime) periodically challenges miners to verify that they have the data they claim to. Since replicas are expensive (slow) to compute, and the challenge window is short, this verifies that the miners in question have not deleted their sealed replicas.

The ongoing compute cost for these PoSts are quite small, I think the current numbers (obviously subject to change) are that you have to do one PoSt per day, which is a single 100M constraint zkSNARK proof, which should take about 10 minutes of machine-with-decent-GPU time to compute.

So I guess for 32GB, it's:

- Sealing: Done once. 3-4h compute on one CPU initially, plus 2-3 hours on a few CPUs.

- Proof of space time: 10 minutes of GPU time per day.

What are the latency expectations within the system when my node is asked for a Proof of SpaceTime? Is it feasible to keep the sealing data, delete the file itself, then spend ~5 minutes downloading the 32GB on my 1 gigabit/sec home Internet connection when someone asks me to do the Proof of SpaceTime, and return the Proof of SpaceTime in 15 minutes (5 min to download, 10 min to compute)?

Thanks again for engaging with me here! It seems the compute time itself isn't a constant enormous cost necessarily.

> Thanks again for engaging with me here!

Of course :)

> What are the latency expectations within the system when my node is asked for a Proof of SpaceTime?

We want them to be as low as possible, but the practicality of getting messages included into a blockchain, and the high cost of running snarks means its definitely on the order of minutes, realistically in the tens of minutes.

A few things to note, the Proof of SpaceTime happens over all of your sectors, not just one. So that ten minutes of GPU time per day is amortized across all of the data you are storing.

> Is it feasible to keep the sealing data, delete the file itself...

Yes, thats feasible if the file you are downloading is already sealed (note: sealing generates a piece of data unique to you, so wherever else that thing is stored is effectively your owned storage). This isnt really different than you having your data stored on a NAS box somewhere. What you cannot do is download the original unsealed data and hope to respond to the challenge in time, as that would require rerunning the expensive sealing process.

I'm very glad I didn't buy into the ICO hype. Filecoin raised $257m selling their Filecoin tokens and now, more than 2 years later, they still have yet to ship a product.

If the market trades Filecoins too high, won't it become a cost burden to those wanting the service? And so then I will leave Filecoin, and there won't much data to store. Until you fix price Filecoin, the technical market will be sullied by the financial markets.

As an example, if my ec2 prices went up in line with Amazon stock, I would be paying 5x the cost compared to z few years ago - which means I would have left to digital ocean, and AWS would lack customers. Lagged hype would falsely bolster the stock price.

It's why this decentralized bullshit doesn't work - in the end the greed of it's founders/majority holders destroy it.

Its not like 1 filecoin == 1 gigabyte of storage. Its a market, people pay what they want to pay. I think you have an incorrect picture of the system in your head and havent actually gone and done the research needed to back up your angry sounding claims.

They pay in Filecoin, which is a fluctuating asset based on financial market conditions which have nothing to do with actual cost of services. Do your research, else your just anther naive investor, or worse, con man. https://tokeneconomy.co/the-analysis-filecoin-doesnt-want-yo...

Lol, whyrusleeping is one of the core contributors to Filecoin.

Also if the price externally is changing, you could just... reduce the amount of FIL you pay per GB.

So if in t=0, the price of FIL is $1, and I choose to pay 1 FIL for 1 GB, and in t=1 the price of FIL goes to $2, I could just pay 0.5 FIL. Sounds like a miner (whose cost basis denominated in fiat) would be happy to accept either deal if its net profitable.

Filecoin's Proof of Replication sounds similar to Valueflo.ws and Holochain's asset backed currencies. Let's go full distributed computing and use mutual credit while we're at it. Otherwise we're just going to carry forward the broken dynamics and assumptions of the current economy.

What is "Mutual Credit"?

Mutual credit means everybody can extend credit to another peer, and there is no centralized issuance of credit. The agreements themselves are fully p2p without any 3rd party. Using mutual credit (which the Holochain pattern/framework is specifically designed to do) [1] also means the mutual credit protocol/agreement itself can evolve without being bottlenecked by the developments of said 3rd party.

"Check out, for example, mutual credit systems like Sardex, where small businesses get together to issue money as credit to each other. Everyone starts from zero in the system, and then goes into and out of short-term debt to one another by either receiving goods and services — which creates a future obligation owed — or by being prepared to offer goods and services to people on the system, which earns you positive claims on others—otherwise known as money." [2]

[1] https://medium.com/holochain/holochain-reinventing-applicati...

[2] https://howwegettonext.com/the-future-of-money-depends-on-bu...

I hadn't heard much from Filecoin since their ICO years ago. How does Filecoin compare to Sia and Arweave?

Reply on news.ycombinator.com