Hacker News Reader

Comment by notmyname | Hacker News Reader

There is already a 100% open source version of Amazon S3: Openstack swift (http://swift.openstack.org). Swift is in use by many companies, and it is the software that runs Rackspace's Cloud Files product, storing petabytes of data and billions of objects.

Swift's code is available on github (http://github.com/openstack/swift) and devs and users are almost always available in #openstack on freenode. There is a wealth of info available and more available to anyone who asks.

I'm all for encouraging many people to solve large-scale storage problems. However, as others have pointed out, nimbus is claiming to be open without providing much detail.

View on HN

(SpiderOak / Nimbus.io cofounder here)

Thanks for your interest! Nimbus.io will have public git repositories, "developed in the open" before we ever charge money to use the service. We just haven't posted the links yet. :)

We admire OpenStack and Ceph as great examples of open source S3 alternatives. Also, Riak+Luwak isn't protocol-level compatible with S3 but offers similar capabilities and an truly elegant design.

Nimbus.io takes a different approach than the above options in that it focuses on space efficiency using parity instead of replication, allowing the storage of a little more than twice as much data using the same hardware. It's a tradeoff of cost vs. latency. For long term archival storage, while throughput matters greatly, latency less so. That's why the price is $0.06/GB.

Great to hear. I'll look forward to looking at your implementation and exploring the tradeoffs you are making. I'm especially interested in how you solve durability in the face of multiple, simultaneous hardware failures. I'm also quite curious about how you are handling object metadata.

You are absolutely right that these things are greatly dependent on the use case. I'm happy to see other people trying to solve these problems too.

Can you describe your API? Do you have your own? Are you reimplementing the S3 API? REST-ful? xmlrpc? How do you handle authentication and authorization?

Swift is....... woeful.

Just as an example, they store object listings using SQLite databases that were file based replicated between nodes for HA. Thus when you had too many files in one container your performance would sink like a stone. Assuming it was never corrupted/etc...

I'm all for people working in this space though. A monoculture is rarely good for anybody.

woeful seems a little harsh :-)

There is a current workaround for the issue you describe: use many containers. However, there are 2 ways to solve the issue for good. One (the simplest) is to have dedicated hardware for the account and container servers, and provide that hardware with plenty of IOPS. Our testing has shown sustained 400 puts/sec on a billion item container with this kind of deployment. The other solution is to change the code to automatically shard the container (transparent to the client) as it gets big. This is something we (the swift devs) are working on. I hope that it will be done in the next several months, but, of course, a complex feature like this is hard to fit to a predetermined timeline.

You're going to shard a SQLite database into a series of objects to deal with "large" containers?

The idea is to limit each "shard" to some number configurable number of objects, say, 1 million. As the container grows, the db can be split in two and each of the two new pieces can grow. The original container entity keeps an index listing of what each of its "child shards" hold, ie the start and end markers.

There are tricky problems to solve, of course. How do listings work? Will shards ever be collected? What are the performance tradeoffs? How does replication handle shard conflicts?

These issue will be worked out, and it should eliminate the write bottleneck in large containers. (Note that reads are/were never affected by this issue.)

This implementation of container sharding is something that is being evaluated. It may or may not ever make it into swift itself.

But why SQlite? And why file based?

Why don't you guys use a proper distributed database to handle container mappings/etc?

Reply on news.ycombinator.com