notmyname

We just added a CONTRIBUTING file to OpenStack Swift (and I believe the other OpenStack projects are getting one soon): https://github.com/openstack/swift/commit/fdf55c2817c9a457de....

The odd thing about OpenStack (from a GitHub perspective) is that the OpenStack development process doesn't use GitHub's issues or pull requests. This has been documented on the OpenStack wiki, but that info is now also in our CONTRIBUTING file. Our file is an example of something to share with potential contributors to point them in the right direction.

You may be interested in "Made of Code" http://madeofcode.com/posts/29-photo-my-new-textmate-theme-8...

Sample of Made of code theme in SL2: http://d.not.mn/made_of_code_sample.png (the code is https://github.com/openstack/swift/blob/master/swift/proxy/c...)

Ah. my mistake. I completely missed the "'" as I was reading it.

On step 3, where you say "scrypt(s'i, md5(si, password))", don't you actually need "scrypt(s'i, md5(s0, password))", where s0 is the original salt? In other words, you still need to know the original salt you were using to successfully migrate.

Therefore, if you are storing the per-user salt as the first bytes in the hashed password field, then you have to be careful when you "throw away the old weak hash hi and forget it ever existed."

True. Your content needs to be in Cloud Files, not on your own server. The storage and cdn services are tied together into the product. They have not been separated to allow the CDN on top of any arbitrary endpoint.

I don't see the requirement of storing the data in cloud files as a very heavy burden, but I'm not the most unbiased source on that.

Rackspace Cloud Files supports this. The file "test_javascript.js" was saved non-compressed. It works the other way, too (compressed->uncompressed if the client doesn't support compression):

    $ curl -i http://d.not.mn/test_javascript.js
    $ curl -i -H "Accept-Encoding: gzip" http://d.not.mn/test_javascript.js

This isn't trying to take away from their announcement. I'm always impressed by Amazon's ability to rapidly deliver features.

Check out http://www.opensource.org/licenses/category. You can probably find something you like in the "License that are popular and widely used or with strong communities" section.

The basic requirements that you described are probably met by either Apache2 or BSD. Others like GPL meet your requirements but add other requirements as well (eg derivative works must be open as well).

Often times licensing can become a religious debate. Read the major ones and choose the one that is best for you.

Yes, swift provides multiple redundant copies of the data. It depends on what you are protecting against: failure of a piece of your data or failure of a hosting provider.

If you simply rely on "dumb disks" spread across multiple providers to provide availability, then you may be interested in looking at the nova-volumes part of openstack (it provides block storage attached to nova VMs). As part of openstack, it's an open system that is seeing rapid adoption.

However, one of the most-requested features for swift is to provide support for logical clusters that span a wide geographic area. This could potentially allow multiple providers to collaborate in providing a multi-provider storage system. However, I'd guess that the technical problems are much simpler than the business problems in setting up multi-provider clusters.

If this were built on top of (or in conjunction with) Openstack swift (http://swift.openstack.org), it could be done with a common storage backend. There are already several public cloud storage providers using swift (Rackspace, HP, Softlayer, Internap, KT, and others), and a growing number of private deployments as well (Wikipedia probably being the most recognizable name).

If you'd like to talk more about this, send me an email (address in profile).

A coworker of mine wrote a series of posts on how to build a consistent hashing ring and expand it with the concepts or weighted nodes and availability zones.

http://www.tlohg.com/p/building-consistent-hashing-ring.html

Those ideas for the core of the consistent hashing found in Openstack swift (a large-scale distributed object storage system).

French. Or Arabic. Or whatever you don't already speak.

Snark aside, the standard answer is something like "Learn a language in each of the different kinds or programming (functional, procedural, etc)". Really, though, programming is not about a particular syntax. It's about learning how to think and deconstruct problems.

So that brings me back to the spoken languages. Learn new things that will challenge you (spoken languages are great), and learn how to deconstruct problems. And above all, have fun doing it.

Such is the challenge of hosting public content. .rackcdn.com is for content hosted on Rackspace Cloud Files and served via our partnership with Akamai.

Cisco's SenderBase recently changed the rating for .rackcdn.com and Bad Things (tm) happened. We (Rackspace) have been in contact with Cisco, and our reputation with their products has been restored (see http://www.senderbase.org/senderbase_queries/detaildomain?se...). Unfortunately, this probably won't be fixed for you until your security filters update the policies from senderbase itself.

Let me know if you have any other issues with this. john.dickinson@rackspace.com

Great to hear. I'll look forward to looking at your implementation and exploring the tradeoffs you are making. I'm especially interested in how you solve durability in the face of multiple, simultaneous hardware failures. I'm also quite curious about how you are handling object metadata.

You are absolutely right that these things are greatly dependent on the use case. I'm happy to see other people trying to solve these problems too.

Can you describe your API? Do you have your own? Are you reimplementing the S3 API? REST-ful? xmlrpc? How do you handle authentication and authorization?

The idea is to limit each "shard" to some number configurable number of objects, say, 1 million. As the container grows, the db can be split in two and each of the two new pieces can grow. The original container entity keeps an index listing of what each of its "child shards" hold, ie the start and end markers.

There are tricky problems to solve, of course. How do listings work? Will shards ever be collected? What are the performance tradeoffs? How does replication handle shard conflicts?

These issue will be worked out, and it should eliminate the write bottleneck in large containers. (Note that reads are/were never affected by this issue.)

This implementation of container sharding is something that is being evaluated. It may or may not ever make it into swift itself.

woeful seems a little harsh :-)

There is a current workaround for the issue you describe: use many containers. However, there are 2 ways to solve the issue for good. One (the simplest) is to have dedicated hardware for the account and container servers, and provide that hardware with plenty of IOPS. Our testing has shown sustained 400 puts/sec on a billion item container with this kind of deployment. The other solution is to change the code to automatically shard the container (transparent to the client) as it gets big. This is something we (the swift devs) are working on. I hope that it will be done in the next several months, but, of course, a complex feature like this is hard to fit to a predetermined timeline.

From my experience working with Rackspace Cloud Files, customer sizes are all over the map. Some customers are very small. Some are very large. I know that S3 has a similar variance in customer size.

From my experience talking to users (and potential users) of Openstack (http://openstack.org), there again is variance. Most people are relatively small (a few hundred GB to a few hundred TB). Some are much bigger (several PB). The most exciting thing I heard was that CERN is evaluating Openstack swift (http://swift.openstack.org) for their storage needs. A researcher from CERN gave a keynote at the last Openstack design summit. CERN generates 25 PB / year and has a 20 year retention policy. They have vast storage needs. The storage needs vary greatly.

I've seen that outsourcing infrastructure is great to a point, but the largest users can generally get substantial cost savings by bringing their infrastructure back in house.

Actually, many people consider it. Some are simply cautious about hosting their data with a third party. Some are prevented from using a third party for compliance or regulatory reasons. Also, it's generally more cost-effective for extremely large datasets to be self-hosted rather than hosted by a third party.

As others have said, being open source is important because it gives you more control over your data. If you have access to the code that is storing your data, you have the option to host it yourself or pay someone else to host it with the same client. I've written about this more on my blog (http://programmerthoughts.com/openstack/democratization-of-d...).

If you are looking for an alternative to S3, I'd ask that you look at Openstack swift (http://swift.openstack.org). It's 100% open source, proven at scale in production, and, if you are hosting it yourself, can offer lower op-ex than using S3 (of course, there is a cap-ex cost to buy your hardware).

There is already a 100% open source version of Amazon S3: Openstack swift (http://swift.openstack.org). Swift is in use by many companies, and it is the software that runs Rackspace's Cloud Files product, storing petabytes of data and billions of objects.

Swift's code is available on github (http://github.com/openstack/swift) and devs and users are almost always available in #openstack on freenode. There is a wealth of info available and more available to anyone who asks.

I'm all for encouraging many people to solve large-scale storage problems. However, as others have pointed out, nimbus is claiming to be open without providing much detail.

Some of the hardware choices that Backblaze has made in their pod design are interesting. Most of the concerns I have (like the difficulty of getting to drives to replace them) can be addressed by operational practices (update the swift ring rather than immediately replace the drive). Other concerns (like 2gbe for 135TB) are more nuanced. Lack of redundancy in the Backblaze pods is addressed by swift itself--swift will ensure that no two copies are on the same pod.

I would love to see someone run swift on some Backblaze pods. If you'd like to talk further, my contact info is in my profile (or drop by #openstack on freenode).

As we were developing openstack swift, we evaluated many different file systems and settled on xfs because our testing showed it to be faster than other file systems when storing many small files (where "many" is much more than the number of files in the linux kernel).

Unfortunately, I can't find the test results that back up my claim.

how does this compare with webob (http://docs.webob.org/en/latest/index.html)?

Completely agree.

When deploying Rackspace Cloud Files (Openstack swift), we tested several SSL-terminating load balancers (throughput, not connection/sec), and we found the best performance with Zeus. Pound was the best in our tests for open source.

Copied from serverfault:

Since you are looking for storage numbers, I'm assuming you are referring to swift, openstack's object storage system.

Swift by default stores 3 replicas of the data. This means that total storage / 3 is your billable or usable storage. The number of replicas is configurable on a cluster level.

However, you can get more detailed. I'd recommend you keep your drives at less than, say, 80% capacity to allow for sudden bursts of storage needs and to give you time to acquire new hardware and grow your cluster. Also, keep in mind that every 2TB the manufacturer advertises is only 1.82TB. Then you have some overhead for formatting.

The above gives you "total raw storage" * 0.90949470177292824 * .8 / "replica count" = "billable storage".

Plugging in your numbers, we get 96TB * 0.90 * .8 / 3 = 23.28TB billable.

Power consumption and cooling will, of course, be very dependent on your chosen hardware and hosting (DC).

Yes, swift can handle heterogenous drive sizes. For example, you can start with 2T drives and start adding 3T drives when they become more cost effective. You initial 40-60TB is well within reason for a swift cluster. I know of several running clusters at much larger scale than that (PBs of data, billions of objects).

Cost consideration are also highly dependent on your particular hardware choice. We recommend that you optimize your hardware for price per GB rather than for IOps or RAM or CPU. As you can imagine, there are nuances to all of this, too.

In case you haven't seen it yet, all the auto-generated docs for swift are at http://swift.openstack.org and much help can be found in #openstack on freenode IRC.

I'd recommend that you first look at the swift all-in-one docs (http://swift.openstack.org/development_saio.html) for running an entire cluster in a single VM. It should give you a good feel of the different parts of the system.

I'm going to lean towards swift, but then again, I'm a core dev for swift and I haven't played with riak/luwak.

Swift is ideal for storing static content (backups, web resources, documents). It's designed to be very scalable (both with concurrency and with total storage space). And it's designed to work with commodity hardware (read: cheap), so it handles failures well (ensuring that data is replicated and safe from bit rot). Also, swift has basic S3 compatibility support.

Please let me know if you have other, more specific questions, and I'll try to answer to the best I can.

Both examples worked with Red Laser on my phone, but the "pretty-fied" one took quite a bit longer for the app to recognize.

I'm a dev for cloud files at Rackspace.

This is something that we are getting with Akamai. Content will be able to be served compressed or uncompressed based on the Accept-Encoding header. So you will be able to store your uncompressed content in Cloud Files and serve it compressed to users whose clients send the Accept-Encoding: gzip header.

since this link requires a login to see the content, try http://news.ycombinator.com/item?id=2096103