Sia – Decentralized cloud storage network

One of the main innovations of Sia is providing a low cost means to turn that 1 TB refurbished hard drive into a useful cloud product.

Sia is able to provide, at 1/10th the cost of Amazon, high throughput, high reliability, high redundancy, geographically distributed cloud storage, and it can do that without high reliability drives or datacenters.

Off topic, but: I purchased three refurb 6TB HGST He HDDs in Jan. from a reputable shop. All three failed last month. I was so happy that the shop completely refunded all of them because the warranty is generally very limited on refurb HDDs. Normal RMA is 5yr on this specific model.

There's no way refurb drives are cost effective for these environments.

> What a refurbished hard disk does not give you: [...]

The whole point is that a decentralized storage network can add such features on top of the 1TB refurbished hard drive.

The refurbished drive may not be reliable, but if you replicate your data amongst many separate drives with independent failure (geographically distributed; different providing entities) then you can create a system with greater reliability. By using erasure coding schemes (similar to RAID5/6), it's possible to reduce the storage overhead which supports reliability to a fraction of the principle amount. Thus, providing 1 TB of storage to the network could earn slightly less than 1 TB of reliable storage in return.

Comparatively, a single cloud provider has the disadvantage that, because they are a single legal entity, they can decide for whatever reason to not continue your service.

Appreciate your response. My comparison to the cost of a hard drive today is not meant to be a direct comparison, but instead to demonstrate that traditional cloud storage is actually quite expensive. Sia is able to bring down storage costs by an order of magnitude and bandwidth costs by two orders of magnitude. While most HN readers understand the nuances of cloud storage costs, most non-technical readers are unfamiliar with the high costs of S3 storage and egress.

Obviously you're paying for more than the drive. That doesn't make it a bad comparison. There's no great reason for the drive to be more than half the cost, so even with 3 copies amazon still comes out as making a whole lot of profit. There's a reason that multiple competitors can provide all those things at significantly lower prices.

And more importantly, if you're focused on storage then don't pay for features you don't need.

I think the use-case of cloud storage is mostly marketing by cloud provides to show relevance to the consumer. They are are loss leaders at best; used to lock you in to their services / track your every move; at worst.

I think the comparison with a $20 1TB hdd is perfect. Cloud providers, and phone makers, have convinced you to have your photos, music, data, online, always. Like a bank. Among other things, its much easier to mine your usage patterns that way and enforce DRM of other cloud services.

I have my IDE PC drive from 20 years ago. It still works. I can burn a CD if im that worried about failure. I dont need the redundancy of a billion user website, a networking api, access control, all that other crap you use in enterprises that comply with ISO standards.

Before cloud providers, people didnt take their photos to Apple and tell them to store them. The only reason u do it now, is because of marketing and service integration.

Decentralized computing is a challenge because unlike storage, it's very expensive to encrypt computation. You see something like 10,000x slowdowns.

With storage, we can encrypt and checksum everything client-side in a cryptographically secure manner, and we can have full faith that even though we are giving our data to someone else, they have no way to view the data or tamper with the data.

If techniques like homomorphic encryption gain substantial speedups, I'm sure that we'll start to see decentralized/trustless computing, but for now you really need to be doing your outsourced computing with someone you can trust and someone who is regulated, because it's possible to act maliciously.

Storj was a brilliant idea then suddenly someone put blockchain into it.

Decentralized computing requires redundant computations for verification which seems like a waste for most applications and not to mention latency added to each request.

I'm sure there's use cases for this that require more sophisticated compute than what smart contract platforms like Ethereum offer but I'm not sure what those are. If someone is being prohibited from using a cloud platform and needs to resort to a decentralized hosting platform, I can see them simply using a VPN and hosting the application on a different platform that's not censoring them. SONM requiring KYC for their decentralized platform sounds like an oxymoron.

I would also add – give Sia a try, especially if you have tried it before our v1.4.0/v1.4.1 releases! https://sia.tech/get-started

Pretty arrogant to say you're going to make cloud storage providers obsolete when you haven't even built a competing solution. You can't even share data yet...

And what about all the other features that cloud providers offer? You say "[the] Sia software is exponentially improving, and performance and featureset is quickly approaching Amazon S3", but I see absolutely no evidence of even the most basic features of cloud storage providers.

Authorization? Regulatory compliance? High-performance bandwidth? High-reliability public API/URLs? Customer support? Private cloud peering? Zero effort integration with many (most?) major data-related open source projects? Versioning? One-click, no-knowledge reliability?

How much do you need to grow to hit cloud scale? 1,000,000x? More? And you're already claiming that you're going to make S3 obsolete?

Your blog would hold more weight if you weren't so over-the-top, bullshit-level optimistic.

Gotta say, the fact that the installation instructions consist primarily of getting around the "unknown / unidentified developer" restriction doesn't inspire confidence.

Hi,

Any chances that Sia will eventually support directly restic/Borg?

I did not know about filebase (and looks interesting), but honestly it is barely competitive with backblaze/wasabi for backups.

The implication is that with Sia you can get those same benefits without having to essentially pay the entire cost of a new hard drive every month. (Sia's rates are much cheaper.)

Aren't all those costs priced in? I assumed the data storage with Sia was redundant and encrypted, which should cover basically all of those things, right?

Looks like it from looking at the site:

>File segments are created using a technology called Reed-Solomon erasure coding, commonly used in CDs and DVDs. Erasure coding allows Sia to divide files in a redundant manner, where any 10 of 30 segments can fully recover a user's files.

[...]

>Before leaving a renter's computer, each file segment is encrypted. This ensures that hosts only store encrypted segments of user data.

I'd love for someone to do an analysis on the performance and availability of the data. I wonder if you could pay more for faster speeds or higher levels of redundancy? That would be neat features.

I think the pricing information should probably be updated. The current public page for Google One says $99/year for 2 TB.

> I would expect that building large datacenters and employing people to manage them is exactly how you go about providing reliable storage capacity.

Further, I imagine that if there is money to be made providing reliable storage to Sia users, then those with experience of reliably providing low-cost, high-volume storage will corner the market pretty fast!

Sia is built on a completely different assumption than most large datacenters - uptime requirements for hosts are around 95%, which equates to about 36 hours of downtime per month.

This means that you can run a much leaner database and require substantially less expertise to keep things going. If a rack goes down, you don't need someone on-site to bring it up ASAP, you can fix it the next workday without upsetting your customers.

95% uptime for hosts translates to 99.99+% uptime for Sia users, because users store data across hosts in a 10-of-30 scheme. As long as 10 out of 30 of the hosts are online, you will be able to access your data. The probability of losing 20 hosts when each of your 30 hosts have 95% uptime is exceedingly small; your practical uptime depends more on the reliability of the software than it does the reliability of the hosts you use.

As for commoditization: that's the goal! If we can completely commoditize data storage, prices should come down dramatically, and the market should be a lot more efficient. You won't need to be an Amazon or Google to have a stable and competitive data offering.

Sia has come a long way in the past two years. Uploads, downloads, and repairs are both much more stable and also a lot faster. Sia can effortlessly sustain 100mbps upload and download if your connection is that fast.

Sia is happy with files that are hundreds of GBs in size today, and with a filesystem that that gets up to about 20 TB. Beyond that it starts to struggle, but we're continuously working to expand its capabilities.

Data has always been encrypted client-side, even in very early releases. Data today is sharded between hosts in a 10-of-30 Reed-Solomon scheme.

I think you will find that the experience is much smoother and more complete compared to the experience from 2 years ago. And, a critical feature is now available as of the latest release: seed based file recovery. After you upload your files, it's possible to create a snapshot of your data that you can recover later on a different machine using nothing more than your wallet seed. This makes Sia a practical and cost effective solution for secondary or tertiary backup.

> Weird caveat I found: the smallest chargeable size is actually pretty large.

This and the cost to form contracts is is large enough that it doesn't make economical sense until you are storing > $1gb.

That said, they seem to slowly be working through a number of these concerns. They very recently released seed based backups, one of the biggest issues I had last time I looked.

Yes, Filecoin is trying to come up with a "more efficient" and possibly "elegant" implementation based on algorithms that are still being invented. Of course, with 100x the budget it's possible that they could still overtake Sia.

Most 4K video is closer to 50mbps. If your home connection is that fast, Sia should be more than capable of delivering smooth 4K streaming to your home.

A single host with slow storage is not a blocking factor because Sia downloads in parallel from as many as 30 hosts at once when composing a stream of data. Even if each has an I/O bottleneck of 1 MB/s (8mbps), you'll be able to get your video out at 240mbps.

In practice today, Sia hosts are almost all network bottlenecked, most use HDDs which do not have trouble keeping up with network latencies (HDD seek times are 11ms, network latencies usually 50ms+) or throughputs (HDDs can usually get close to 100 MB/s, far more than the network throughput)

Can I ask, have you tried streaming 4K video from the latest version of Sia? Seek times might be 5-10 seconds, but I believe that once your stream has loaded you will see completely smooth playback. Our next release has a focus on reducing those seek times, but the code is not ready yet.

You are correct that the failure rate for contracts is somewhere around 9%, and you are also correct that you need over 10 contracts to get 11 9's of redundancy, however this doesn't translate to needing 10x redundancy on the Sia network to get high reliability.

Data is uploaded to Sia (per-default, it's configurable through the API) in a 10-of-30 scheme using Reed-Solomon coding, which means that each piece of data is held by 30 hosts, and out of those 30 hosts any 10 of them are sufficient to recover the original data. This has a total overhead of 3x, and the algorithms behind it are in my opinion super fascinating.

If you assume each host independently has 91% uptime, you get this amount of downtime as a result: https://www.wolframalpha.com/input/?i=sum+of+(30+choose+x)+*...

In reality, the software reliability is a bigger factor in downtime than the host's reliability.

Non-reliant on central services or entities.

There are a lot of details such as finding servers with available space, paying for storage, posting bonds, collecting bonds with fraud proofs, and doing all of that in a decentralized way. You pretty much need a blockchain for the decentralization.

You need the blockchain to secure venture capital investment and a vocal online community.

Founder here. The blockchain gets you a couple of things.

The key thing in our blockchain is the storage contract, which is something like a blockchain-SLA. The host has an obligation to store some data, and the host also puts up money out of pocket as a promise that the data will be stored faithfully. The blockchain will occasionally challenge the host, and if the host cannot create a proof that the data is still being stored, the host's collateral is forfeit.

The renter also puts money into the file contract, this money is used to pay the host if the host is honest/keeps the data, and the money is destroyed if the host is dishonest or loses the data. The money gets destroyed so that the renter has no reason to interfere with the host, nothing to gain from seeing the host fail or working to make the host fail.

Thanks to the blockchain, the host has a guaranteed payment if they store the data, regardless of whether the renter sticks around for the duration of the contract or not. And also thanks to the blockchain, the renter knows that the host will not get paid until the end of the contract, and that they will actually lose some of their own money if they lose the data.

There's another great benefit though to cryptocurrency, which is that payments are super low friction. Every time you upload data to the Sia network or download data from the Sia network, you make hundreds to thousands of tiny payments to the various parties over state channels. This wouldn't be feasible using more traditional payment systems, we'd have to track everything in a centralized database, and that actually doesn't scale very well.

Sia is both low cost and high speed, but these are side benefits over the primary goal of uncompromising decentralization. Storage just happens to be an application where the decentralized version is actually highly competitive to the centralized versions.

I'm glossing over a fair amount here, if there's anything that doesn't make sense or seems off feel free to ask, I'm happy to clarify.

> that just syncs other people's stuff instead of yours and pays you monthly.

with VC money? with a fickle settlement system that requires 5 layers of financial institutions before you get a REST API? where you probably still need 50+ money services licenses and compliance department anyway? where any of those financial institutions can still cut you off for any reason? to maintain operations in a single country?

why do all that when you can just print money, or more accurately create a scarce digital asset of value to the market you create, available internationally on day one

you don't have to agree with it, but the pendulum has swung waaay in this other direction and thats the answer for "why" before you even get into the implementation that the founder wrote.

Can you please clarify which CDN features are you referring to on your blog (can't find any) or anywhere else? Also can you elaborate what do you even mean by "so well"? Sia was started to be built before IPFS was even announced and has hosted PBs of data already as a working product. On the other hand IPFS hasn't even released it's public testnet so it would be great to understand your point. Edit: btw by IPFS I assume you refer to their Sia equivalent product, Filecoin. At least I do, as IPFS cannot be compared to Sia alone.

I believe that censorship resistance is a key value proposition for decentralized file storage. So its agnostic handling of illegal content is a feature, not a bug.

All data is fragmented and encrypted, so it's impossible to know whether a particular host is storing a fragment of illegal content or not. I suppose if you somehow _did_ find out you were hosting a fragment of illegal content you could breach the storage contract (and pay the resulting penalty) but that's very unlikely because, again, all content is encrypted.

It doesn't. The last real application of Sia that I heard of during the previous crypto boom basically sounded like renting your HD space out to people who would potentially use it to store illegal content.

I find the concept of "Illegal Content" conceptually questionable. Authorities do not care much, I assume, if you possess illegal content. What they care about is if you share illegal content.

Say you have a pro-democracy pamphlet on your hard-drive in China. If they find it they would imprison you no doubt but they don't care so much to look for it, and it is hard to find if you never share it with anybody.

SIA provides private storage so if your only crime is possessing illegal content but never sharing it with anybody no big deal (I would assume). But if you share it then SIA can not protect you because the exchange must happen somewhere outside of SIA.

How does http handle illegal content? Is there any reason that content should be blocked at a protocol level instead of punishing the person who created it through existing regulations? If Sia has a way to remove such content it loses its decentralized nature, which is the whole reason it exists in the first place.

Should it exist? Well sometimes important content is illegal, such as political content in China. It’s good to have a way to publish important illegal content, even if bad stuff slips through.

What if someone posts illegal violent or sexual content? That is unfortunate, but it means that a real person took the time to post such content on the network. Keeping them from speaking will not change their warped beliefs, it will just hide them from the world and keep people from remembering that evil people exist. The person who published the content should be punished through existing regulations and the content should stay on the network to remind us that some people are horrible. Hopefully no one accesses it.

All data is encrypted automatically before it is uploaded to the network. It is impossible for individual hosting nodes to see what is being stored.

You could store your Sia seed for free on Google Drive. That way you get the benefit of Google being "liable" for your data and credentials, but with the much lower storage costs of Sia. Yes, it's a centralized third party, but you already implied you have no problem with that.

Sia's seeds are ridiculous-- the 29 words provide 300 bits of entropy. 100 bits would be a sufficient security margin against brute forcing, assuming a modern memory-hard KDF like Argon2.

With a 100 bit password, assuming every flop of the 1.8 exaflops of the Top500 supercomputers tested a new password, it would still take 25,000 years to crack. Key stretching should add at least 30 bits of security by taking a billion operations--

    $ perf stat argon2 asdfasdf -id -m 16 -t 16 <<< asdf
    Type:  Argon2id
    Iterations: 16 
    Memory:  65536 KiB
    0.781 seconds

         6,331,020,712      cycles                    #    4.021 GHz                    
        13,467,211,117      instructions              #    2.13  insn per cycle         

Here's what 100 bits of security margin looks like with a more sophisticated scheme (abbrase): "Hope raised between unpleasant bellows. Devil rode sullenly, refugees waiting." => (first three letters) hopraibetunpbeldevrodsulrefwai.

They explain what the mean by that two sentences later:

> Hosts on Sia do not have to worry about building and maintaining enormous datacenters, employing thousands of employees, and marketing their services. They only need to worry about providing reliable storage capacity to renters on Sia.

And yes, it's essentially a decentralized storage market built on smart contracts. You can either pay to store your files (the client software handles redundancy, contracts, etc automatically) or _get_ paid to rent out space on your hard drive.

I think they are saying they are removing "trusted" intermediaries. There are zero trusted intermediaries sitting between buyer and storage/bandwidth in Sia's model. Thus, value cant be extracted from providing trust/confidence.

I agree that the use cases don't seem to be lining up. So is this for personal use or corporate data center replacement? Consumer companies will give you 2TB for $10 with no bandwidth costs with guaranteed replication and sharding built in. What happens when one of these nodes goes down? I didn't see anything in the post about that. I think the thing that always gets lost in these ideas is that there is a reason that people pay these current prices. It is because is solves a real problem and high reliability requires paying people to make sure that is true. A smart contract and collecting micro payments on the hope a cryptocurrency will rise in price have much different skin in the game.

I've always thought the most important use case around decentralized storage was more around censorship. You can post content that is harder to block in countries that firewall AWS, etc.

coming from someone who is not a fan of Siacoin from when I last used it in 2017, I think you miss the point

what I like and want is encrypted distributed data that I can store and retrieve quickly and cheaply

I also like an economic model that aligns the hosts to maintain the data, and how the proliferation of that economic model lets me invest in its growth from merely being a passive non-productive speculator, and finally I would like a clear easy to understand ROI from being a productive host: can I subsidize the cost of my hardware that I'm already interested in having.

any cryptocurrency can do that, but how the software uses the cryptocurrency is a key factor, and unfortunately we can like an idea and team but get forced to accept an economic model that has flaws

so for me, it is easy for me to understand why teams want to promise the sky against S3, and it really isn't that relevant to using this securely. like, you might be able to subsidize all of your hardware and provide a service without worrying about if the SIA comparison is accurate because its really not relevant.

but finally, it turns out that unmonetized and non-cryptocurrency based decentralized cloud storage networks are good enough! so I stick with IPFS.

hope that was useful to someone.

I agree in that they are probably comparing the wrong aspects of S3.

You're not going to be able to compete on the speed anytime soon probably, and maybe even the capacity if it really takes off. But, anyone here that works on an Amazon-based internet company knows that S3 is the most unreliable (in terms of availability, not storage) universally used product they have, next to EC2.

A good chunk of incidents can be attributed to S3 downtime in some AZ. With a decentralized storage network, the probability of actual downtime in terms of availability go wayyyyy down. If Amazon us-east-1 has a bad deploy or a network event, you lose access to those buckets, full-stop.

With something like this, you'd need a much larger network event, and if the app is fully decentralized itself, blast radius can be contained to individual customer regions instead of where you host your servers.

Probably more useful to compare to GCS Storage honestly, since they have multi-region buckets.

You are being deceived by Amazon's PR machine at its finest. The 11 9s of durability are a "design", not a guarantee. The official PR line at the time was "S3 is designed to provide 11 9s of durability".

There's still no SLA on durability AFAIK.

> Please stop adding layers and layers of calculations, it's a waste of energy.

The trend in electronics is that this has become less true over time. As transistors have shrunk, the relative cost of calculation has diminished compared to the cost of transmission over distances. A cryptographic operation can be cheap compared to accessing main memory because the distances signals travel within the crypto logic block are tiny.

Complexity (as in more to understand) does not equate sensibly to higher power consumption.

From what I understand, right now these projects are working on the backbone of the systems. In the future, other developers will be able to create services on top of these platforms to resell the storage space. So maybe someone will have an online cloud hosting service with a simple drag and drop online interface. They will charge a fee to the customer and the website operator will simply buy storage contracts on SIA or similar providers to provide the actual hosting and distribution of the files.